Mean/Std curves from Lesson 10

Part 2 (2019)
1

A big part of lesson 10 was the running means and stds of the outputs of each layer. They can be seen in the following notebook to be very erratic for 2 epochs:https://github.com/fastai/course-v3/tree/master/nbs/dl2 as:

21%20PM
Screen Shot 2019-09-07 at 11.35.21 PM.jpg994Ɨ1086 179 KB

Then we saw that if we just saw for the first 10 batches they were flat:

00%20PM
Screen Shot 2019-09-07 at 11.36.00 PM.png940Ɨ1014 92.6 KB

Then we tried our version or ReLu and batchnorm to show how these can be made stable, however when I trained a model on fashion MNIST I got pretty flat statistics right from the start for 12 epochs. I ran the same model and got:

40%20PM
Screen Shot 2019-09-07 at 11.33.40 PM.png1386Ɨ944 148 KB

I dont understand why Iā€™m not getting the different curves as Jearmy did

2

Update: Turns out the problem only happens when learning rate is high. Thus the lesson motivation was what happens to training when the lr is too high

1 Like