Anyone else having an issue where the after the model is finetuned to make predictions on the dogs vs cats redux data, the predictions are binary? When I generate predictions on dogs/cats data with the vgg model before fine tuning, I get fractional probabilities for each of the 1000 imagenet categories (eg. .23 likelihood of Egyptian cat). However, after finetuning, I never get any predictions with fractional probabilities between 0 and 1, they’re always exactly 0 or exactly 1. The reason I ask is because the scoring function on kaggle is more forgiving of incorrect predictions closer to .5, eg for a given example, .55 likelihood of dog, .45 of cat, rather than 1 of dog, 0 of cat.
1 Like
I’m having the same issue, but in addition it’s assigning classes of goldfish and tench to the images instead of dog and cat. These are the first two classes in the original vgg/imagenet model, so I guess Keras didn’t pull the names from the training directory structure?
I can totally replace the labels, but I get the idea I’ve missed something.
@tom / @jbrown81 please let us know exactly what code you’re using, and what results you get - some tips on asking for help are here: http://wiki.fast.ai/index.php/How_to_ask_for_Help . Once we have this info, I’m sure we’ll be able to resolve your problems.
Can I reflect back what I think the todo list is for lesson 1?
- Get AWS instance running (either g2 or m4 if not yet approved for g2) after contacting support etc.
- Setup ssh keys as per instructions in setup video
- install bash setup script onto server instance
- launch jupyter notebook on the instance
- once the notebook is running, review the lesson 1 notebook notes and run each cell of code to figure out what python and vgg is doing
- install kaggle CLI onto the server instance
- use the kaggle CLI to download the current data for the Dogs vs. Cats Redux competition
- configure the new data to the file structure in the same way that was used in the sample lesson 1 notebook
- make a copy of the lesson 1 notebook and use the new copy to draw in the new Dogs Vs. Cats data (including moving utils.py and vgg16.py to the new folder where the new notebook sits?)
- Run the relevant code cells on the sample set of new Dogs v. Cats data to make a prediction on the new image data set.
- Once, the sample set works, modify the jupyter notebook to use the on the new test data images
- write a script that takes the predict() data of the new Dogs vs. Cats data and writes the data to a new csv file in the format of the sample_submission.csv file that was downloaded with the Dogs vs. Cats
- submit that new submission.csv file to the kaggle via the CLI tool
- check the public scoreboard for your own ranking
- modify or tune current code in the lesson 1 notebook to try to get into the top 50% ranking of the current Dogs v Cats competition
- start exploring the other new datasets on kaggle and decide which one you or some teammates would like to study further during the course
- download the new data to your EC2 instance and repeat the previous steps with your brand new data.
Is this about right?
I understand that the first lesson is mostly about getting comfortable with a terminal to manipulate a server instance in the cloud, how to organize raw data, and how to participate in the kaggle community.
Are we also learning how to fine tune the model onto the data? the examples with the VGG() model seem already optimized with batch=64. To submit to the kaggle competition did you mean for us to optimize the “the Create a VGG model from scratch in Keras” with the new Dogs v Cats data in order to get above 50% in the kaggle ranking?
22 Likes
At this stage of the class, what knobs could we be using for fine-tuning VGG16 for dogs vs cats and scoring “well” (within top half of leaderboard) in the Kaggle competition?
- running more epochs?
- tweaking the optimizer parameters? (not really covered yet…)
** I wanted to add that I was able to submit and get on the board but a little bit shy of the top half 
I think discussion about how to tune the model is happening here:
1 Like
If anyone is looking for documentation on the methods that a VGG object takes:
https://keras.io/models/model/
Could be helpful in tuning the prediction or understanding the parameters and returns of methods like predict()
4 Likes
hi @leahob. same here, i was a bit shy of the second half after submitting (running more epochs seemed to improve the score, didn’t have a chance to fiddle with the optimizer parameters much)
this happened to me on some runs, but not on others (e.g. for some values of num_epochs but not for others)
1 Like
What was your validation set accuracy? It’s easier to make suggestions once we know how you’re going so far.
Actually I can give you a strong hint - look at the equation here: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation . Have a think about what minor change to your output might make a big difference to that evaluation function. Hint: I went from 105th spot to 37th spot by running two little commands in my text editor - no need to even start up my AWS instance…
2 Likes
Thanks @jeremy
First, I ran the vgg model as is (with 1000 class output, without fine-tuning to dogsvscatsredux) to generate predictions on the first 4 test images like so:
from vgg16 import Vgg16
vgg = Vgg16()
batches = vgg.get_batches(path+‘test’, batch_size=4,shuffle=False)
imgs,labels = next(batches)
vgg.predict(imgs, True)
and the output I get is:
(array([ 0.2321, 0.5742, 0.2567, 0.5104], dtype=float32),
array([285, 246, 229, 285]),
[u’Egyptian_cat’, u’Great_Dane’, u’Old_English_sheepdog’, u’Egyptian_cat’])
Next, I fine tuned the vgg model to generate two-class predictions like so:
vgg = Vgg16()
batch_size=128
batches = vgg.get_batches(path+‘train’, batch_size=batch_size)
val_batches = vgg.get_batches(path+‘valid’, batch_size=batch_size)
vgg.finetune(batches)
vgg.fit(batches, val_batches, nb_epoch=1)
training completed, then I ran prediction on the test set:
batches = vgg.get_batches(path+‘test’, batch_size=4,shuffle=False)
imgs,labels = next(batches)
vgg.predict(imgs, True)
and the output I get is:
(array([ 1., 1., 1., 1.], dtype=float32),
array([0, 1, 1, 0]),
[u’tench’, u’goldfish’, u’goldfish’, u’tench’])
The predictions on the first 4 test images look correct (cat,dog,dog,cat). What I’m puzzled by is why the probabilities are always exactly 1. With a softmax output, I expect the class probability values be somewhere between 0-1, like they are with the original vgg net.
I’m running this on a p2 instance fwiw.
4 Likes
Can confirm this worked
I submitted after only just one epoch of training to test the mechanics of submission after seeing that the validation accuracy is not too bad.
Thanks for the hint below regarding the evaluation details, Jeremy – noting the type of prediction probabilities I submitted (mainly 0, 1 as noted also by @jbrown81), and the evaluation metric, there is definitely room for quick improvement.
Epoch 1/1
22500/22500 [==============================] - 629s - loss: 0.7596 - acc: 0.9504 - val_loss: 0.2705 - val_acc: 0.9824
1 Like
@jbrown81 we’ll talk more about that issue in class, but the short answer is that the model tends to be terribly over-confident; you can’t really think of them as probabilities at all. So it might well be giving ones and zeros as ‘probabilities’. If you try running predict() before you train, you should see that you do indeed get numbers between zero and one.
help debugging writing to csv. I thought I was iterating through the batches to get the predictions and the filename ids at the same time, but the file ids are repeating with different prediction scores associated with them. I think it’s iterating through the images and predicting ok, but I’m not sure how to get beyond the first 64 item batch to the next.?
Does batches.filenames seems to return the whole list [whether 1k or 12k items] of all files in the target directory or is it only the one batch size at a time? I thought it was the former, and then batches get divvied up by next() before being passed to predictions.
any help would be appreciated.
Also, i’m training my model on the m4.instance and it’s been going for about 4 hours and it still isn’t done. As a reference, training models would be be faster on the on p2 instance?
def pred_batch():
import csv
# collect batches for analysis
batches = get_batches(data_path+'train', batch_size=64, shuffle=False)
# open csv to write to
with open('kaggle_catsdogs.csv', 'w') as csvfile:
# assign Kaggle column names
fieldnames = ['id', 'label']
# instaniate DictWriter to write to csv
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
# write colnum names to csv
writer.writeheader()
# while loop to continue loading batches after the first
# batch of 64 elements are analyzed
while next(batches):
# iterate through batches
imgs, labels = next(batches)
# run images through prediction method
preds = vgg.predict(imgs, True)
# index values of images
idxs = np.argmax(preds, axis=1)
#loop to format predictions and files id
for i in range(len(idxs)):
idx = idxs[i]
# split to get file id
filename = batches.filenames[i].split('.')[1]
print ('{},{:.1f}'.format(filename, preds[i, idx]))
writer.writerow({'id': filename, 'label': preds[i, idx]})
2 Likes
@melissa.fabros yes .filenames gives you the whole lot. If you come into class now we can take a look and help you out.