I have set up my personal workstation like following:
HOST: macbook pro 2017(intel i5 2.3GHz | 8GB memory)
GPU: Nvidia GTX 1080 (using bizonBox )
OS: MAC OS High Sierra 10.13.2
Virtual Environment:
- Keras 1.10
- python 2.7
- theano 1.0.1
- pygpu 0.7.0
- numpy 1.13.3
- scipy 1.0.0
(not sure if need more information)
the issue occurs when I run following code in lesson 1
whenever I training the model nearly 55% before finishing. My GPU will be powered off, I have used a system monitor app to check the usage of GPU, it has reached 100%. I thought it might be caused by overloading. However, I have done a GPU pressure test, there was no problem when GPU cores usage reaching 100%. This issue really annoy me a lot. Could anyone help me figure it out?
I have copied code from jupyter-notebook to a separate python file like following:
import theano
batch_size=32
import vgg16; reload(vgg16)
from vgg16 import Vgg16
vgg = Vgg16()
path = "./data/dogscats/"
# Grab a few images at a time for training and validation.
# NB: They must be in subdirectories named based on their category
batches = vgg.get_batches(path+'train', batch_size=batch_size)
val_batches = vgg.get_batches(path+'valid', batch_size=batch_size*2)
vgg.finetune(batches)
vgg.fit(batches, val_batches, nb_epoch=1)
and it runs okey, it seems jupyter-notebook caused the GPU powered off, I cannot find any reason for that.
anyone had the same experience?
have solved this issue, it caused by the jupyter-notebook progressbar, using this progress bar instead. Then change same code in utils.py and lesson1 notebook like following
def fit(self, batches, val_batches, nb_epoch=1, verbose=0, callbacks=[]):
"""
Fits the model on data yielded batch-by-batch by a Python generator.
See Keras documentation: https://keras.io/models/model/
"""
self.model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=nb_epoch,
validation_data=val_batches, nb_val_samples=val_batches.nb_sample, verbose=verbose, callbacks=callbacks)
notice here I only add two extra arguments(verbose, callbacks)
in jupyter notebook add following:
from keras_tqdm import TQDMNotebookCallback
# change this method
vgg.fit(batches, val_batches, nb_epoch=1, verbose=0, callbacks=[TQDMNotebookCallback()])