Averaging saved models

If I use the SaveModelCallback to save N models, is there a way I can average the weights of the N saved models before using learn.load()?

Hello, I’m also new to fastai (and fastai2) but I think you can use it with fastai, the functionality is called SWA.

Like this here in pure pytorch (from the original authors):


So in fastai I saw here in these files in the fastai repo (found in the old part specifically):
/fastai/fastai/old/fastai/swa.py
/fastai/fastai/old/fastai/model.py
fastai/fastai/old/fastai/learner.py

In fastai2, it’s not yet there (as far as I saw).

So you can find the Learner object has fit_gen() and it has these params:
use_swa=False, swa_start=1, swa_eval_freq=5
I think just flip use_swa=True, gives what epoch you want to start saving and which intervals
(as far as I know you should only use the last ‘models’ at the end of learning, those are good weights based on the paper)

Hope it helped you a bit.

2 Likes