Pytorch training loop vs Fastai Learner

Part 1 (2020)
3

What kind of problem are you working on?

There are various defaults that fastai has that leads to improved performance. Here are some things to check (somewhat specific to pretrained image models):

  • The optimizer used (fastai uses AdamW)
  • The hyperparameters used (fastai has different defaults for betas, weight decay, etc.)
  • Custom model modification by fastai (ex: fastai adds a custom head when fine-tuning pretrained image models)
  • One-cycle actually starts decreasing 25% into training (pct_start default). Other defaults to be aware of include the starting and ending LR for one-cycle.
  • One-cycle also schedules the momentum as well

There are probably a lot that I am missing but these are the main ones I can think of (and see in the code).

4 Likes