Lesson 1 discussion

(Louis) #579

To see if it’s overfitting compare this accuracy with the validation set accuracy which is printed when the epoch is done. The difference should not be to big. For example if with your results, val_accuracy is 0.80 you are overfitting.

To save the model use

I use this in a call in the notebook so I don’t know how to do it after every epoch in the code. But if I wanted to do it, I’d first check to see if it can be added to the callbacks= parameter in model.fit_generator. (But maybe I am dead wrong!)

(Slava) #580

Hello Jeremy, Rachel. Thank you very much for such wonderful course!
I found that retyping the code on my own really works - start understanding things.
I stucked a little on line:
imgs, labels = next(batches)
So I drilled down to gen.flow_from_directory function, looked up keras documentation. However still cant figure out this line of code. What it does?
Thank you!

(Louis) #581

It returns the next batch of images with their labels. So if your batch size was defined as 5, it returns 5 images and their one hot encoded labels vector.

 def get_batches(self, path, gen=image.ImageDataGenerator(), shuffle=True, batch_size=8, class_mode='categorical'):
            Takes the path to a directory, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
            See Keras documentation: https://keras.io/preprocessing/image/
        return gen.flow_from_directory(path, target_size=(224,224),
                class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)

This is where it is all setup. “gen” is an ImageDataGenerator object. When get_batches() returns gen.flow_from_directory(...) it is assigned to the variable batches upon which you then iterate by calling next(). Each time you call next() you get batch_size number of images and batch_size number of one hot encoded vectors which represents the category of the image (dog or cat). You can loop indefinitely upon the ImageDataGenerator and it will always keep returning 4 images.

All this is used to endlessly loop over the training set during training.

Now there are many ways to get data out of an ImageDataGenerator and flow_from_directory is one of them. You can check the other ways in the ImageDataGenerator documentation.

Hope this helps!


(Louis) #582

Slava, see my reply just above. (pressed the wrong reply button)

(Slava) #583

Thank you a lot Louis, now I got it. Very helpful! :+1:

(Hugo) #584

Have you figured out how to iterate to the next batch? I am running into the same problem.

(Louis) #585

in fit(), model.fit_generator automaticallly calls batches until it has run the specified number of epochs.

(Hugo) #586

Thanks for the quick reply Louis. I looked into the vgg16.py file and decided to use the function “vgg.test” instead of “vgg.predict”, which calls the predict_generator instead of flow_from_directory. Jeremy said in the lecture that prediction can be done in batches also, I’m curious on how that works. If we are predicting the test set without labels, shouldn’t we use predict_generator instead of fit_generator?

(Louis) #587

predict_generator() requires a generator objects and predict() a numpy array. So both are ok depending on what you want to pass along.

But check carefully: predict_generator() uses test_batches which comes from get_batches() which returns a generator (test_batches) which is getting it’s data from flow_from_directory. :wink:

(Alessa Bandrabur) #588


I have the following error. I want to load the already saved weights and test on the test_path which contains 12500 pictures. And I have the following error.

If I run exactly the same code on valid_path it will work
batches, preds = vgg.test(valid_path, batch_size = batch_size*2)
Found 2000 images belonging to 2 classes.

I don’t know what I am doing wrong … I really appreciate your help.


I would guess that in your test_path, you didn’t create a pseudo class for test data.
Both train and validation data have sub-directories for classes, i.e. cats and dogs. All images are under the subdirectories.
I think the get_batches() function assume such structure when it fetches data.
To solve the issue, try create such directory:
And put all images under it.


I’m trying to install git after ssh’ing to an aws instance:

However, I get the following error when running sudo apt-get install git:

buntu@ip-10-0-0-6:~$ sudo apt-get install git
│Reading package lists… Done
│Building dependency tree
│Reading state information… Done
│The following extra packages will be installed:
│ git-man liberror-perl
│Suggested packages:
│ git-daemon-run git-daemon-sysvinit git-doc git-el git-email git-gui gitk
│ gitweb git-arch git-bzr git-cvs git-mediawiki git-svn
│The following NEW packages will be installed:
│ git git-man liberror-perl
│0 upgraded, 3 newly installed, 0 to remove and 53 not upgraded.
│Need to get 3,285 kB/3,306 kB of archives.
│After this operation, 21.9 MB of additional disk space will be used.
│Do you want to continue? [Y/n] y
│Err http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main git-man
│ all 1:1.9.1-1ubuntu0.3
│ 404 Not Found [IP: 80]
│Err http://security.ubuntu.com/ubuntu/ trusty-security/main git-man all 1:1.9.1
│ 404 Not Found [IP: 80]
│Err http://security.ubuntu.com/ubuntu/ trusty-security/main git amd64 1:1.9.1-1
│ 404 Not Found [IP: 80]
│E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git-man_1.
│9.1-1ubuntu0.3_all.deb 404 Not Found [IP: 80]

│E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git_1.9.1-
│1ubuntu0.3_amd64.deb 404 Not Found [IP: 80]

│E: Unable to fetch some archives, maybe run apt-get update or try with --fix-mi

(shicheng) #593

Hi, Jeremy

appreciate if you could explain a little bit about VGG16, what is 16 referring to?


Hi Jeremy,

Appreciate for the lesson, may I ask a question about how to import the class.

It seems that in the code in lesson 1, class Vgg16 are imported by

import vgg16; reload(vgg16)
from vgg16 import Vgg16

However, is it better to import the code in this method? And if the former one is better, why should we use the method?

from vgg16 import Vgg16

I don’t which method is better. I really appreciate your help.

(Aditya) #595

Refer the original paper…
They are like

 'VGG16, VGG19'

(Erik Chan) #596

It would help a lot if this thread was broken into a separate folder. I think many students are having different questions relating to lesson 1 and this thread is nearly impossible to follow because of it.

(Erik Chan) #597

I’ve been running into the exact same issues as you have. It should have been more obvious “/test/” was necessary. I am now also running into the problem where its not clear which image (filename) is being predicted

(Erik Chan) #598

I can’t figure out what I’m doing wrong.

My predict function predict = vgg.predict(imgs, True) for the test images outputs three arrays:

  1. The first array is the predict confidence for the each test image
  2. The second array are 1 and 0s, I believe it is referring the whether the image is a cat or dog
  3. The third array are labels ‘cats’ or ‘dogs’ (which is the same thing as the second array).

Isn’t my predict function supposed to output a confidence level per category?
I am expecting three arrays but each array element has two values (one pertaining to dogs category and the other pertaining to cats category), otherwise how would I receive the confidence level just for dogs to submit to kaggle?


thanks louismg


Maybe the comment from function predict() can help:

            preds (np.array) : Highest confidence value of the predictions for each image.
            idxs (np.ndarray): Class index of the predictions with the max confidence.
            classes (list)   : Class labels of the predictions with the max confidence.