I decided to run some experiments to see how each hyperparameter/optimizer interacts together. Here’s what theta0 and theta1 look like across 10 epochs on unsorted data:
model = MySimpleModel()
learn = Learner(dls, model, loss_func=loss_function_MSE, lr=1, opt_func=partial(Adam, mom=0.9, sqr_mom=0.75, wd=0, decouple_wd=False), cbs=TestCallback, train_bn=False, moms=(0.95, 0.85, 0.95))
learn.fit(10)
learn.opt.state_dict()
And here are 10 epochs on sorted data:
model = MySimpleModel()
learn = Learner(dls, model, loss_func=loss_function_MSE, lr=1, opt_func=partial(Adam, mom=0.9, sqr_mom=0.75, wd=0, decouple_wd=False), cbs=TestCallback, train_bn=False, moms=(0.95, 0.85, 0.95))
learn.fit(10)
learn.opt.state_dict()
Here is a gist of the behavior: https://gist.github.com/kevinbird15/a3aa9f9d3fc9607086c67532d8751040