Learning Rate Finder for small datasets

Part 1 (2018)
1

Using personal data set of 700 images. Does the learning rate finder not work on smaller datasets?
How can I debug this issue.

image
image.png1014×549 25.2 KB

image
image.png980×294 9.28 KB

3 Likes
2

How big is your batch size? If it is too small you’ll get exactly what you are seeing.

3

Using the default value for batch size, in this case 64.
Should it be lower?

Trying for 32, will post update.
Is there any rule of thumb for identifying bs value for a small dataset.

4

no smaller can lead to the problem you are seeing.

are you running on CPU or GPU? I had a similar problem running on my CPU a ways back.

of course, the size of your training set may still be the problem. if it is, one idea is to duplicate your examples X # of times to artificially create a much bigger dataset. or you can use some data augmentation to do this as well.

5

I also had this problem. I’m using a very small data-set (25). When I ran the lr_find with the default batch size (64). I got a linear graph and blank graph for learn.sched.plot, so I started reducing the bs [16,8,4,2] - when I got to 4, I was able to see a graph for learn.sched.plot but not a nice looking graph, then I changed bs to 2 and I got a plot that doesn’t decrease nicely like what @jeremy had - I’m not sure of what point to pick from this graph. I think this finding is interesting and I would love to learn more about all these different results from changing bs

  1. bs 64, had a pretty low accuracy of 54.5% but training and validation loss are not that far apart. learn.sched.plot was blank (why please?)
  2. bs 32 has a very good accuracy of about 80.7% but the validation loss is a far lower than training loss (what does this mean please).
  3. bs 16 had way lower accuracy 48.6% with a very high validation loss compared to training loss.
  4. bs 8 had a very high accuracy of 81% but insane validation loss.
  5. bs 4 I picked lr of 0.01 and got accuracy of 79% after two epoch
  6. bs 2, result fairly close to No 5.

I would like to have better understanding about the results I got.
Thanks.

links to screenshots








1 Like
8

Look at Lesson 1: my experiments with resnet34 and some questions

Sooner or later, the fastai staff will address the issue. I guess they just forgot. I’ll try and open another specific thread about that.

9

hey guys, I built an image classifier for minerals and wrote my first medium post based on it, please check it out and let me know your thoughts