Very high validation loss and unstable training when using adam optimizer (Notebook 09c)

Part 2 (2019)
1

hi! when playing around with the lessons notebooks I noticed that using adam leads to very unstable training and high losses in notebook 09c, see e.g. the image below

image
image.png773×303 8.46 KB

As far as I can see thing work fine in notebook 09 (where adam is first introduced) and also in 09b.
The only change I did in 09c before running the notebook was to change the optimizer to adam in the last few lines of the notebook i.e. to change this

learn = get_learner(nfs, data, 0.4, conv_layer, cb_funcs=cbfs)

into this

learn = get_learner(nfs, data, 0.4, conv_layer, cb_funcs=cbfs,opt_func=adam_opt())

has anyone experienced the same issue?

2

so newbie mistake from my side, leaving a comment here for other people running into the same: is of course not possible to take any random notebook and just replace the optimizer to use adam instead of other optimizers without also adjusting the learning rate.

so the issue here was that the default lr used through the course is usually too high for adam, so one need to make sure to use something like 0.001

1 Like