Hiho!
I’m trying to use discriminative learning rates for a progressively growing auto encoder.
That means I’m basically growing an auto encoder from the inside out, starting with low res images up to high res. Now I want to reduce the learning rates for the inner layers once I increase my resolution (and thus add more convs in front of the encoder and at the end of the decoder).
As far as I understand the implementation the following things happen:
- A learners
learn.layer_groupsis used as a base to determine parts of the networks that have the same lr. Thus, there will belen(learn.layer_groups)different learning rates. -
Learner.lr_range()is what sets thelearn.lrand it basically makes annp.arraywith the same length aslayer_groups. I’m assuming that the LRs are then mapped onto those likezip(layer_groups, lr)…
The thing I did not find yet is where exactly the optimizer gets to know of all this.
My question now is: If I want to change the discriminative LR behavior to what I describe at the top, is it sufficient to change the Learner.lr_range() function to return an array that correctly maps my learning rates? And how can I best confirm that all the layers are actually trained with the correct learning rate then?
Greetings,
Dome
