Lesson 2 discussion

balnazzar · November 13, 2017, 3:56pm

Hi! Apologies in advance if my question is inappropriate or if this is not the place where it should be posted.

Anyway, I’m referring to statefarm-sample.ipynb, where a validation set and a sample are created for the State Farm dataset (which comprises just train and test in its vanilla version).

The notebook begins by entering the train directory, then creates the appropriate directories. Note that at no point a valid/ folder is created in the dataset root (rather it is only created in sample/).
Then, the script generated the sample set by shuffling train, picking 1500 imgs, and copying them into sample/train.

Immediately afterwards (we still are in dataset-root/train/), the following magic command is issued:

%cd …/valid/

But no such folder exists, since the original dataset didn’t have a validation dir.

So, I gather I have NOT to reshuffle my ‘g’. Instead, I should just pick the following 1000 images, create a dataset-root/valid/ folder, and copy those images in it.

Am I right?

gnavink · November 15, 2017, 7:54am

Hi,

I was following the notebook of lesson2. I could not understand certain code areas. attaching the screen shots:

Here I believe val_classes is assigned the classes of the validation batch val_batches. Its intuitive…
But where can I get the documentation that the attributes of the object val_batches has??

Similarly
1.png984×208 16.6 KB

model.fit_generator uses the arguments samples_per_epoch = batches.n .
How can i know that batches.n is what should be assigned to sample_per_epoch?

I tried to see the keras documentation mentioned in the FastAI wiki page (http://wiki.fast.ai/index.php/Main_Page pointing to
https://faroit.github.io/keras-docs/1.1.1/

I tried looking in https://faroit.github.io/keras-docs/1.1.1/preprocessing/image/ . Even though that page is a great tutorial of how to use those API’s I couldn’t get the info what I wanted…

Even though the tab operator of jupyter notebook gives options of different attributes I do not know how to get info what those represent.

any ideas on this…

thanks.
navin

callum.yap · November 15, 2017, 11:29pm

Hey fam, I’m running into this error when I try to create my validation and sample set. Does anyone know how to fix it? Thanks

nzhang · November 16, 2017, 2:52am

get_batches() calls image.ImageDataGenerator().flow_from_directory() as in vgg16.py, which returns a
DirectoryIterator object. And you can see classes is one of its attributes.
Check file: keras/preprocessing/image.py in keras github.

You can find fit_generator() models.py in keras/models.py code as well, and you can see what input arguments are needed.

gnavink · November 16, 2017, 7:23am

thanks @nzhang . Seen the files suggested by you. got the info needed.

Need to dig into the code with some knowledge of python to decipher what the attributes of the DirectoryIterator object contains…

I believe there is no documentation yet in keras website where I can find what each attribute of the DirectoryIterator represent… Am i right??
By the way, the keras documentation is excellent…

thanks
navin

prettycold · November 16, 2017, 4:17pm

Hi,

I did AWS setup as told by Jeremy and initially used a t2.micro instance to get a hang of AWS.

I was able to successfully execute Lesson1 code in my local laptop itself. I was able to run vgg.test without any hassles. Later moved to a p2.xlarge instance. Tried to execute Dogs Vs Cats Redux and do a kaggle submission. Am unable to go past batches, preds = vgg.test(test_path, batch_size = 64).

It gives me the following error:
imageimage.png1116x268 12.3 KB
I took utils.py, vgg16.py, vgg16bn.py from github. For whatever reasons, copy link address and wget saved .py files as .html. So I did a cat > utils.py and pasted contents from git. I’d missed the line return test_batches in vgg.test method initially. I figured that out and corrected. But no use.

I initially ran 3 epochs, but since I wanted to get this up and running, I changed to 1 epoch and re-executed. Noticed below in ft0.h5 under results folder

imageimage.png1116x268 12.3 KB
But I don’t see any error or warning message in notebook for that cell execution. Am sure something is wrong.

Went through all possible discussions in Lesson1 and Lesson2 forum, but looks like none has faced this.

Would somebody be able to help me ? Dying to make my 1st kaggle submission Thanks in advance.

Unknowingly, opened a new discussion topic. Below is the link which has snapshots of my error messages.

nzhang · November 17, 2017, 12:41am

I am not aware of documentation for attributes of the DirectoryIterator. Source code is good enough

nzhang · November 17, 2017, 12:43am

Might be directory setup. Your images should be under:
test_path/unknown/cat*.jpg

prettycold · November 17, 2017, 8:37am

@nzhang,
Yeah, they were in the right directory.

I re-started my kernel as @gnavink suggested and am able to pass through vgg.test. Probably something with respect to path got messed up as I was executing cells multiple times.

Thanks!

gnavink · November 18, 2017, 6:25am

@npvisual

Hello nicolas, looked at your notebook to avoid memory problem in Lesson2 because of get_data(). its quite good that you have tried it on a different note… would have saved me a day’s effort if I had seen it before. I also stumbled upon the same approach which you had taken that of calling model.predict_generator().

Is there any reason why you had made shuffle=False while calling get_batches()

I get 97% validation accuracy as expected if shuffle=False. But when i call get_batches() with shuffle= True both my training & validation accuracy drops down to around 50%.

I have no clue why it happens… My understanding is shuffle=True on a training data should provide randomness to have a better accuracy while training the model… But I don’t know why the accuracy drops down…

Quoted the code here for better understanding of the problem…

`Preformatted text

batch_size = 64

train_batches = get_batches(trainpath, batch_size = batch_size, shuffle = False )
valid_batches = get_batches(validpath, batch_size = batch_size, shuffle = False )

def onehot(x): return np.array(OneHotEncoder().fit_transform(x.reshape(-1,1)).todense())

train_labels = onehot(train_batches.classes)
valid_labels = onehot(valid_batches.classes)

vgg = Vgg16()
model = vgg.model

train_features = model.predict_generator(train_batches, val_samples = train_batches.N)
valid_features = model.predict_generator(valid_batches, val_samples = valid_batches.N)

lm = Sequential([ Dense( 2, activation = ‘softmax’, input_shape=(1000,)) ])
lm.compile(optimizer = RMSprop(lr = 0.1), loss = ‘categorical_crossentropy’, metrics = [‘accuracy’] )

lm.fit(train_features, train_labels, batch_size = batch_size, nb_epoch = 3,
validation_data = (valid_features, valid_labels))
`
I get a training & validation accuracy of around 97% after training 3 epochs.

Now restarted the kernel. Made shuffle = True and re-ran the model.

indent preformatted text by 4 spaces

train_batches = get_batches(trainpath, batch_size = batch_size, shuffle = True )
valid_batches = get_batches(validpath, batch_size = batch_size, shuffle = True )

Any suggestions on why such a drop in accuracy??

navin

gnavink · November 18, 2017, 7:16am

My query was regarding the post by nicholas
http://forums.fast.ai/t/lesson-2-discussion/161/271

Trying to quote that post here using quote post feature…
hope it helps to correlate the query with that specific post.

I apologize in case the query is quite messy… New to this forum . trying to learn how to use quote post feature of the forum.

gnavink · November 18, 2017, 11:02am

Is there any reason why :

In Cell No:14 val_batch with 23000 samples is used while doing model.predict_generator() instead of train_batches. I believe you want to get the 1000 category predictions for training data in that cell.
Also the batch_size has been set to 1 when using get_batches() for training data while valid_batches has batch_size of 40. I hope its just a choice …

npvisual · November 18, 2017, 5:28pm

Navin,

Yeah, if I recall correctly (haven’t looked at this notebook recently though) if you use shuffle = TRUE then your batch set won’t match with the actual labels you’re feeding to the model.

The reason for that is that you’re shuffling the images (random order) and then you’re training the model against an ordered list of labels.

It actually makes sense that your accuracy would drop to 50% : the model at this point is probably just recognizing cats and dogs as being the same and guessing correctly half of the time (totally random).

HTH,
N.

npvisual · November 18, 2017, 6:31pm

on #1 : I think you’re right. This should be train_batches. Probably just a copy & paste error from cell 11. I will re-run that part to see if it changes anything. My guess is that I already had the training predictions on disk and when I reformatted the notebook it used that.

on #2 : you should definitely change that. I was originally doing some tests to see if there was a significant difference in processing time and memory consumption when using different batch sizes. I probably forgot to reset it to using batch_size=40.

If I have time next week I’ll update the notebook to Keras 2. Since I updated my environment, it’s a little difficult to revert back to Keras 1.2.

Thanks !
N.

gnavink · November 20, 2017, 7:50am

was doing the lesson2 notebook ie removing the last dense layer of 1000 predictions of vgg’s model
and adding a dense layer with 2 outputs with dogscats data. The same dogscats data was also used in the class. I got the validation accuracy to 97.75 %. But when I plot the confusion matrix i get weird results…
I assume that when val_accuracy is around 97% and the validation data is 2000 samples, the no.of misclassifications should be only around : (3/100) * 2000 = 60
But my confusion matrix is showing:

The number of misclassifications here is: 486 + 488 =974
Not sure why its high when val_accuracy is 97%.

My code is as follows:

indent preformatted text by 4 spaces

vgg = Vgg16()
model = vgg.model

train_batches = get_batches(trainpath, batch_size = batch_size)
valid_batches = get_batches(validpath, batch_size = batch_size)

Found 23000 images belonging to 2 classes.
Found 2000 images belonging to 2 classes.

model.pop()
for layer in model.layers:
layer.trainable = False
model.add(Dense(2, activation = ‘softmax’))
model.compile(optimizer = RMSprop(lr = 0.1), loss = ‘categorical_crossentropy’, metrics = [‘accuracy’])

def fit_model(train_batches, valid_batches, nb_epoch = 1 ):
model.fit_generator(train_batches, samples_per_epoch=train_batches.N, nb_epoch = nb_epoch,
validation_data = valid_batches, nb_val_samples = valid_batches.N)

fit_model(train_batches, valid_batches, nb_epoch = 1 )
Epoch 1/1
23000/23000 [==============================] - 628s - loss: 0.6630 - acc: 0.9565 - val_loss: 0.5950 - val_acc: 0.9610

fit_model(train_batches, valid_batches, nb_epoch = 1 )
Epoch 1/1
23000/23000 [==============================] - 628s - loss: 0.5184 - acc: 0.9672 - val_loss: 0.3477 - val_acc: 0.9775

probs = model.predict_generator(valid_batches, val_samples = valid_batches.N)
preds = np_utils.probas_to_classes(probs)
cm = confusion_matrix(valid_batches.classes, preds)
plot_confusion_matrix(cm, valid_batches.class_indices)

any thoughts why its so…

navin

Sohrab · December 7, 2017, 9:26pm

I have been going through lesson 2 part one and I am not sure whether I’ve identify and issue in the original code in “Training multiple layers in Keras” section.

I am not sure if we are updating weights if “model.compile” has not been run after updatating the first Dense layer to a trainable layer. So basically, without compiling the model after updating the layer (or layers) to become trainable, running more epochs will not update the corresponding weights. Although looking at the summary of the model, it shows that the weights are trainable but underneath the model is not updating the weights. In keras2 this issue will raise a warning but in Keras1 no warning will be flagged. I run the model on two system with Kears 1 and 2.

To elaborate on this issue I have ran few test and in the following I am sharing my results on Keras2. Image below shows the process of dropping the last layer, adding a dense layer with 2 output and running two epochs to train this last layer. As you can see each epoch takes about 107s and 592ms/step.

Image below shows the result of changing the first dense layer to trainable but not compiling the model and Keras 2 is raining a waning. very similar epoch and step speed to the previous section.

After compiling the model, the loss increased from 0.08 to 8.06 and we see a slightly longer processing time.

Furthermore according the Keras documentation, one must compile for the effect to take place:

“Additionally, you can set the trainable property of a layer to True or False after instantiation. For this to take effect, you will need to call compile() on your model after modifying the trainable property
” (https://keras.io/getting-started/faq/)

So in conclusion, I believe in the section that we tired to train and update the weights on the last Dense layer we are not doing anything. Please let me know if I am missing something here?

Here are the links to my notebooks:

Keras 1:

gist.github.com

https://gist.github.com/srsani/544adf217c9e3c8aa542a8b10ae6c698

leason2_srs_keras1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {

This file has been truncated. show original

keras 2:

gist.github.com

https://gist.github.com/srsani/85376349b2dfdede966c2f5ba23f143d

SRS_leason2_keras2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {

This file has been truncated. show original

unnik · December 27, 2017, 11:02am

Following this post. Same question.

unnik · December 27, 2017, 11:35am

Hi,

Facing same issue. Did you figure out why?

mhutson · January 7, 2018, 8:54pm

It looks like the dog and cat images have a mixture of sizes:
http://wiki.fast.ai/index.php/Lesson_2_Notes#Visualization_of_Results
How does the convnet handle input matrices of different sizes?

nomothetis · January 8, 2018, 10:58pm

Hi all.

I’ve run into something I think is strange. I’m following the steps recommended in the Lesson 2 notebook, namely:

Replace the last layer and train it with all other layers fixed.
Make all layers after and including the first Dense layer trainable, and train again.

What I’ve observed, consistently, is that the training loss goes from 1.x (1.29 in my latest attempt) back to 2.3 (i.e. random) when I start training step 2. Is this normal? Does anyone know what the loss would jump back up?

Thanks.