If I use the SaveModelCallback to save N models, is there a way I can average the weights of the N saved models before using learn.load()?
Hello, I’m also new to fastai (and fastai2) but I think you can use it with fastai, the functionality is called SWA.
Like this here in pure pytorch (from the original authors):
So in fastai I saw here in these files in the fastai repo (found in the old part specifically):
/fastai/fastai/old/fastai/swa.py
/fastai/fastai/old/fastai/model.py
fastai/fastai/old/fastai/learner.py
In fastai2, it’s not yet there (as far as I saw).
So you can find the Learner object has fit_gen() and it has these params:
use_swa=False, swa_start=1, swa_eval_freq=5
I think just flip use_swa=True, gives what epoch you want to start saving and which intervals
(as far as I know you should only use the last ‘models’ at the end of learning, those are good weights based on the paper)
Hope it helped you a bit.
2 Likes
in fastai2, it’s not yet there (as far as I saw).
any updates on this topic, I am looking for SWA in fastai. I see some implementations of SWA but not sure how to make use of that with learner object with cbs. also swa was merged in older fastai but could not find that on current fastai code