Lesson1: about the training time of dogscats_redux

###My desktop PC is I7 7700k ,32gb DDR4 and Asus Nvidia1070.

When I run the Finetuning and Training part , the calculation cost me more than 2 hours.

I just feel something strange about the training time.
Is it too long to run only 3 epoch?

##Here’s the running situation when the time passed 2hours.
Running epoch: 0 Epoch 1/1 8386/8386 [==============================] - 4809s - loss: 0.3307 - acc: 0.9703 - val_loss: 0.2331 - val_acc: 0.9840 Running epoch: 1 Epoch 1/1 8322/8386 [============================>.] - ETA: 26s - loss: 0.2918 - acc: 0.9790 ...

The specific line codes are below shown.

for epoch in range(no_of_epochs): print "Running epoch: %d" % epoch vgg.fit(batches, val_batches, nb_epoch=1) latest_weights_filename = 'ft%d.h5' % epoch vgg.model.save_weights(results_path+latest_weights_filename) print "Completed %s fit operations" % no_of_epochs

I think there is some problem in your configuration. I guess your computation is happening on CPU . I haven’t ran on 1070, but on K80 it takes no more than 10 min for 1 epoch. On 1070 it should take even less time, close to 5 mins.

I am not sure about the solution for this. Check other threads, some people have faced the same problem before.

Yeah, something seems to be wrong with your settings. I have almost the exact same setup and I just ran this and each epoch completed in 356s.

When you run the cell that contains

#Allow relative imports to directories above lesson1/
sys.path.insert(1, os.path.join(sys.path[0], '..'))

#import modules
from utils import *
from vgg16 import Vgg16

#Instantiate plotting tool
#In Jupyter notebooks, you will need to run this command before doing any plotting
%matplotlib inline

What is your output? You should see something about your 1070, like this:

Using gpu device 0: GeForce GTX 1070 (CNMeM is disabled, cuDNN 5105)

If you don’t, make sure you have a .theanorc file in your home directory. It should contain something like this:

[global]
floatX = float32
device = gpu

[nvcc]
fastmath = True

Note that it says gpu instead of cpu.

Okay.When I ran the codes below, the feedback shown in the next line is only Using Theano backend.

#Allow relative imports to directories above lesson1/
sys.path.insert(1, os.path.join(sys.path[0], '..'))

#import modules
from utils import *
from vgg16 import Vgg16

#Instantiate plotting tool
#In Jupyter notebooks, you will need to run this command before doing any plotting
%matplotlib inline

And also I have tried this config(Keras 1.22.),but it doesn’t work.
I have started nvidia-docker container before already.

{
"epsilon": 1e-07, 
"floatx": "float32", 
"image_data_format": "channels_first", 
"backend": "theano",
"image_dim_ordering": "th"
}

I try to find out my .theanorc file in my nvidia-docker envirment.

But I cant.

I built the docker envirment based on Ubuntu16.04 with TF on nvidia-docker .
Then I enter the below codes into the bash line.

sudo nvidia-docker exec -ti container_name /bin/bash

.theanorc file is not existed in home directory.

SO I build one like this
root@514698e33bc6:~# vim ~/.theanorc

When I enter

[global]
floatX = float32
device = gpu
[nvcc]
fastmath = True

Another error appeared what made me confused.
What can I do next?

Using Theano backend.

ImportErrorTraceback (most recent call last)
<ipython-input-10-f725a731952c> in <module>()
      3 from glob import glob
      4 import numpy as np
----> 5 from vgg16 import Vgg16# Import our class, and instantiate
      6 from vgg16bn import Vgg16BN
      7 

/notebooks/tfv_data/vgg16.py in <module>()
      7 from scipy.ndimage.interpolation import zoom
      8 
----> 9 from keras.utils.data_utils import get_file
     10 from keras import backend as K
     11 from keras.layers.normalization import BatchNormalization

/usr/local/lib/python2.7/dist-packages/keras/__init__.py in <module>()
      1 from __future__ import absolute_import
----> 2 from . import backend
      3 from . import datasets
      4 from . import engine
      5 from . import layers

/usr/local/lib/python2.7/dist-packages/keras/backend/__init__.py in <module>()
     62 if _BACKEND == 'theano':
     63     sys.stderr.write('Using Theano backend.\n')
---> 64     from .theano_backend import *
     65 elif _BACKEND == 'tensorflow':
     66     sys.stderr.write('Using TensorFlow backend.\n')

/usr/local/lib/python2.7/dist-packages/keras/backend/theano_backend.py in <module>()
----> 1 import theano
      2 from theano import tensor as T
      3 from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
      4 from theano.tensor.signal import pool
      5 from theano.tensor.nnet import conv3d2d

/usr/local/lib/python2.7/dist-packages/theano/__init__.py in <module>()
     64     object2, utils)
     65 
---> 66 from theano.compile import (
     67     SymbolicInput, In,
     68     SymbolicOutput, Out,

/usr/local/lib/python2.7/dist-packages/theano/compile/__init__.py in <module>()
     26 from theano.compile.builders import *
     27 
---> 28 from theano.compile.function import function, function_dump

/usr/local/lib/python2.7/dist-packages/theano/compile/function.py in <module>()
     16 import numpy as np
     17 import warnings
---> 18 from theano import compat
     19 
     20 __docformat__ = "restructuredtext en"

ImportError: cannot import name compat

I’m fresh on docker and theano. Thanks for your help!

For the record, on my machine with a GTX 1080 Ti (with TensorFlow backend) an epoch takes about 300s.

Running on a Macbook Pro (no GPU). i7, 16 GB RAM. first epoch still running and ~ 24 hrs. I didn’t know it gets so bad.

something wrong here ??