Lesson 1 discussion

Part 1 (2017)
580

Hello Jeremy, Rachel. Thank you very much for such wonderful course!
I found that retyping the code on my own really works - start understanding things.
I stucked a little on line:
imgs, labels = next(batches)
So I drilled down to gen.flow_from_directory function, looked up keras documentation. However still cant figure out this line of code. What it does?
Thank you!

581

It returns the next batch of images with their labels. So if your batch size was defined as 5, it returns 5 images and their one hot encoded labels vector.

 def get_batches(self, path, gen=image.ImageDataGenerator(), shuffle=True, batch_size=8, class_mode='categorical'):
        """
            Takes the path to a directory, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
            See Keras documentation: https://keras.io/preprocessing/image/
        """
        return gen.flow_from_directory(path, target_size=(224,224),
                class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)

This is where it is all setup. “gen” is an ImageDataGenerator object. When get_batches() returns gen.flow_from_directory(...) it is assigned to the variable batches upon which you then iterate by calling next(). Each time you call next() you get batch_size number of images and batch_size number of one hot encoded vectors which represents the category of the image (dog or cat). You can loop indefinitely upon the ImageDataGenerator and it will always keep returning 4 images.

All this is used to endlessly loop over the training set during training.

Now there are many ways to get data out of an ImageDataGenerator and flow_from_directory is one of them. You can check the other ways in the ImageDataGenerator documentation.

Hope this helps!

Louis

2 Likes
582

Slava, see my reply just above. (pressed the wrong reply button)

1 Like
583

Thank you a lot Louis, now I got it. Very helpful! :+1:

584

Have you figured out how to iterate to the next batch? I am running into the same problem.

585

in fit(), model.fit_generator automaticallly calls batches until it has run the specified number of epochs.

586

Thanks for the quick reply Louis. I looked into the vgg16.py file and decided to use the function “vgg.test” instead of “vgg.predict”, which calls the predict_generator instead of flow_from_directory. Jeremy said in the lecture that prediction can be done in batches also, I’m curious on how that works. If we are predicting the test set without labels, shouldn’t we use predict_generator instead of fit_generator?

587

predict_generator() requires a generator objects and predict() a numpy array. So both are ok depending on what you want to pass along.

But check carefully: predict_generator() uses test_batches which comes from get_batches() which returns a generator (test_batches) which is getting it’s data from flow_from_directory. :wink:

588

Hello,

I have the following error. I want to load the already saved weights and test on the test_path which contains 12500 pictures. And I have the following error.

image
image.png1093×449 62.3 KB

If I run exactly the same code on valid_path it will work
batches, preds = vgg.test(valid_path, batch_size = batch_size*2)
Found 2000 images belonging to 2 classes.

I don’t know what I am doing wrong … I really appreciate your help.

590

I would guess that in your test_path, you didn’t create a pseudo class for test data.
Both train and validation data have sub-directories for classes, i.e. cats and dogs. All images are under the subdirectories.
I think the get_batches() function assume such structure when it fetches data.
To solve the issue, try create such directory:
test/fakeclass/
And put all images under it.

592

I’m trying to install git after ssh’ing to an aws instance:

However, I get the following error when running sudo apt-get install git:

buntu@ip-10-0-0-6:~$ sudo apt-get install git
│Reading package lists… Done
│Building dependency tree
│Reading state information… Done
│The following extra packages will be installed:
│ git-man liberror-perl
│Suggested packages:
│ git-daemon-run git-daemon-sysvinit git-doc git-el git-email git-gui gitk
│ gitweb git-arch git-bzr git-cvs git-mediawiki git-svn
│The following NEW packages will be installed:
│ git git-man liberror-perl
│0 upgraded, 3 newly installed, 0 to remove and 53 not upgraded.
│Need to get 3,285 kB/3,306 kB of archives.
│After this operation, 21.9 MB of additional disk space will be used.
│Do you want to continue? [Y/n] y
│Err http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main git-man
│ all 1:1.9.1-1ubuntu0.3
│ 404 Not Found [IP: 34.212.136.213 80]
│Err http://security.ubuntu.com/ubuntu/ trusty-security/main git-man all 1:1.9.1
│-1ubuntu0.3
│ 404 Not Found [IP: 91.189.88.152 80]
│Err http://security.ubuntu.com/ubuntu/ trusty-security/main git amd64 1:1.9.1-1
│ubuntu0.3
│ 404 Not Found [IP: 91.189.88.152 80]
│E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git-man_1.
│9.1-1ubuntu0.3_all.deb 404 Not Found [IP: 91.189.88.152 80]
│
│E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git_1.9.1-
│1ubuntu0.3_amd64.deb 404 Not Found [IP: 91.189.88.152 80]
│
│E: Unable to fetch some archives, maybe run apt-get update or try with --fix-mi
│ssing?

593

Hi, Jeremy

appreciate if you could explain a little bit about VGG16, what is 16 referring to?

594

Hi Jeremy,

Appreciate for the lesson, may I ask a question about how to import the class.

It seems that in the code in lesson 1, class Vgg16 are imported by

import vgg16; reload(vgg16)
from vgg16 import Vgg16

However, is it better to import the code in this method? And if the former one is better, why should we use the method?

from vgg16 import Vgg16

I don’t which method is better. I really appreciate your help.

595

Refer the original paper…
They are like

 'VGG16, VGG19'
596

It would help a lot if this thread was broken into a separate folder. I think many students are having different questions relating to lesson 1 and this thread is nearly impossible to follow because of it.

1 Like
597

I’ve been running into the exact same issues as you have. It should have been more obvious “/test/” was necessary. I am now also running into the problem where its not clear which image (filename) is being predicted

598

I can’t figure out what I’m doing wrong.

My predict function predict = vgg.predict(imgs, True) for the test images outputs three arrays:

  1. The first array is the predict confidence for the each test image
  2. The second array are 1 and 0s, I believe it is referring the whether the image is a cat or dog
  3. The third array are labels ‘cats’ or ‘dogs’ (which is the same thing as the second array).

Isn’t my predict function supposed to output a confidence level per category?
I am expecting three arrays but each array element has two values (one pertaining to dogs category and the other pertaining to cats category), otherwise how would I receive the confidence level just for dogs to submit to kaggle?

600

thanks louismg

601

Maybe the comment from function predict() can help:

        Returns:
            preds (np.array) : Highest confidence value of the predictions for each image.
            idxs (np.ndarray): Class index of the predictions with the max confidence.
            classes (list)   : Class labels of the predictions with the max confidence.
1 Like
602

Thanks @nzhang, it didn’t occur to me to lookup the function definition. Great idea.

It looks like the function doesn’t do what i expect, but I found out via lesson 2 the shortcomings.

1 Like