Training a model from scratch: CIFAR 10

(Jeremy Howard (Admin)) #1

I’ve just pushed a couple of things that the iceberg competition folks in particular will find interesting…

Here’s a notebook showing how to train CIFAR 10 from scratch, which also serves as a foundation for any use of custom networks you may want to try: https://github.com/fastai/fastai/blob/master/courses/dl1/cifar10.ipynb

The notebook uses the CIFAR10-tuned ResNext architecture from here: https://github.com/D-X-Y/ResNeXt-DenseNet . I’ve also made the pretrained weights available here: http://files.fast.ai/models/cifar10_resnext32x32_8.h5 . You can use learn.load() to load them, after creating the learner using the steps shown in the notebook. The data to train it yourself from scratch is here: http://files.fast.ai/data/cifar10.tgz

My guess is that this architecture should work well for the iceberg competition. The pretrained weights may be a little helpful, but I’m not sure either way…

(You could easily change the resnext definition to handle 2 channel input directly BTW. Would be a good exercise.)

28 Likes

(Kerem Turgutlu) #2

Wow ! Thank you so much

0 Likes

(Jeremy Howard (Admin)) #3

Hey @kcturgutlu got something much better now! Try from fastai.models.cifar10.senet import SENet18 and grab the weights from http://files.fast.ai/models/sen_32x32_8.h5 . This is the 'squeeze and excitation network` that won Imagenet this year, trained on Cifar10 to 94.1% accuracy :slight_smile:

It’s also really really fast!!!

19 Likes

(Kerem Turgutlu) #4

So exciting thanks :smiley:

0 Likes

(Gerardo Garcia) #5

ELI5 - Can you elaborate on how to load this model?

0 Likes

(Jeremy Howard (Admin)) #6

Follow the steps in the notebook in the top post. Let us know if you get stuck…

0 Likes

(Kerem Turgutlu) #7

What was the intuition behind increasing image size during training ?

0 Likes

(Devan Govender) #8

@jeremy
Section 4 of the Snapshot Ensembles paper indicates that the standard augmentation technique for the CIFAR datasets is to:

  • zero-pad with 4 pixels on each side

  • randomly crop to produce 32 x 32 images

  • horizontally mirror with probability 0.5

Cell 4 of your notebook does the flipping and padding. Is the cropping also accounted for?

2 Likes

(Jeremy Howard (Admin)) #9

Yes, well spotted! When you pad, it random crops by default :slight_smile:

0 Likes

(Devan Govender) #10

Thank you.

0 Likes

(Jeremy Howard (Admin)) #11

Here’s an example of everything to use the pretrained model for 94% on CIFAR (except for get_data)

5 Likes

(James Requa) #12

This is amazing, thank you!! Not to be greedy but are the ImageNet weights available as well? :grin:

0 Likes

(Kerem Turgutlu) #13

Hi @jeremy I know that we can’t have floating padding and you also recommended me to use original image size as sz. In this case we have 75x75 images and pad = sz//8. Should I change padding or sz ? Which makes more sense and why ? Thank you so much again for the amazing update in the library :slight_smile:

Edit: It works with data = get_data(75, 32) but couldn’t understand how pad 75//8 is allowed.

1 Like

(Chan Sooi Loong) #14

What argument does wds mean?
From the code, it states wds (iterable/float), but I don’t quite get what it does.

learn.load(‘sen_32_32_8’) would get this error. Running on AWS AMI instance.

cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:87

0 Likes

#15

wds - weight decay(s)

3 Likes

(Kerem Turgutlu) #16
1 Like

(Jeremy Howard (Admin)) #17

Nope, but if you feel like training them, I’ll host the weights :slight_smile:

2 Likes

(Gerardo Garcia) #18

@jeremy this model uses 10 different categories.
If I’m trying do the dog breed or the satellite images or any other one.
The number of categories will change.

What is the best way to fix the number of categories here to fit to the new model.
I remember in Keras you pop the last layer and you add a new fully connected layer with the number of categories needed.
fastai and pyorch should have something similar
Can you shed some light :flashlight: here?

1 Like

(Anze Zupanc) #19

@gerardo number of classes is an argument of the resnext29_… methods (by default it’s set to 10). See for example definition of resnext29_16_64(num_classes=10) in fastai/models/cifar10/resnext.py.

0 Likes

#20

fastai will do it for you automagically :slight_smile: The whole popping of the last layer and slapping a layer on top that has the correct dimensionality (aligned with the amount of classes in your dataset)

3 Likes