Lesson 1 vgg16 model training cause Nvidia GPU power off

I have set up my personal workstation like following:
HOST: macbook pro 2017(intel i5 2.3GHz | 8GB memory)
GPU: Nvidia GTX 1080 (using bizonBox )
OS: MAC OS High Sierra 10.13.2
Virtual Environment:

  • Keras 1.10
  • python 2.7
  • theano 1.0.1
  • pygpu 0.7.0
  • numpy 1.13.3
  • scipy 1.0.0
    (not sure if need more information)
    the issue occurs when I run following code in lesson 1


    whenever I training the model nearly 55% before finishing. My GPU will be powered off, I have used a system monitor app to check the usage of GPU, it has reached 100%. I thought it might be caused by overloading. However, I have done a GPU pressure test, there was no problem when GPU cores usage reaching 100%. This issue really annoy me a lot. Could anyone help me figure it out?

I have copied code from jupyter-notebook to a separate python file like following:

import theano
batch_size=32
import vgg16; reload(vgg16)
from vgg16 import Vgg16
vgg = Vgg16()
path = "./data/dogscats/"
# Grab a few images at a time for training and validation.
# NB: They must be in subdirectories named based on their category
batches = vgg.get_batches(path+'train', batch_size=batch_size)
val_batches = vgg.get_batches(path+'valid', batch_size=batch_size*2)
vgg.finetune(batches)
vgg.fit(batches, val_batches, nb_epoch=1)

and it runs okey, it seems jupyter-notebook caused the GPU powered off, I cannot find any reason for that.
anyone had the same experience?

have solved this issue, it caused by the jupyter-notebook progressbar, using this progress bar instead. Then change same code in utils.py and lesson1 notebook like following

    def fit(self, batches, val_batches, nb_epoch=1, verbose=0, callbacks=[]):
        """
            Fits the model on data yielded batch-by-batch by a Python generator.
            See Keras documentation: https://keras.io/models/model/
        """
        self.model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=nb_epoch,
                validation_data=val_batches, nb_val_samples=val_batches.nb_sample, verbose=verbose, callbacks=callbacks)

notice here I only add two extra arguments(verbose, callbacks)
in jupyter notebook add following:

from keras_tqdm import TQDMNotebookCallback
# change this method
vgg.fit(batches, val_batches, nb_epoch=1, verbose=0, callbacks=[TQDMNotebookCallback()])