Do we have SWA in V1?

fastai
1

stochastic weight average

2

Not yet no.

3

I’m working on it.

6 Likes
4

Do you need help with htat?

5

@wdhorton Thanks for working on this, would love to use it once its ready.

6

@wdhorton, Also I have a conceptual question… It’s not possible to combine SWA and 1cycle training right?

I.e. SWA is tied to using cyclical learning rates (CLR), so we have to do multiple cycles (instead of just 1 cycle), and keep running-average of weights at end of each cycle, right?

7

Does there happen to be any updates on this?

8

There is an interesting article on the pytorch blog with some example code for SWA: https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/

I guess, the functionality should be easy to implement with a call back.

If I have a working setup I will post it.