Shedding some light about LR management in fastai

[balnazzar (Andrea de Luca)]

@kushaj

final_div is the argument you need for this. For larger values, the difference is hard to notice, but when you use smaller values for final_div you can see the difference.

But the whole point of final_div would be to make some epochs with a very small learning rate. If you look at Gugger’s summary, he recommends something like one hundredth of the minimum. That is (leaving everything at default) 1/25 * 1/100. If you pass a final div of 2500, the plot should actually show the graph touching the line y=0, but this does not seem to happen.

Also, from your example plot with final div = 2, you can see it alters the whole cycle, not just the final part. Looking at the typical linear example, the final part is the line segment characterized by a slightly smaller negative slope:

The main cycle is the part of the graph above the line y=0.001, the final part is the one below that line. Setting such part should not alter the rest of the cycle.

I would not dare to tag Jeremy, but since he liked the answer, I would just ask him for clarification particularly about such matters.

@kushaj

So for cosine, there is not much sense of specifying the length of the final part, as it only depends on the value of the max_lr and min_lr. Now if you want to use other annealing functions(which you should not)

Mh, why not? You can safely glue a cosine (in fact, since it starts from ~0, I would have called it sine annealing) with another function… It would be continuous nonetheless.

Just another thing: what is the point of having max_lr? It doesn’t seem to do anything useful…

@adeperio

So I think if you setup a cnn_learner with a pre-trained network, you would get 2 layer groups by default. Assuming we have an unfrozen network, if you passed a slice to max_lr with two values, you would get one learning rate applied to the first layer group (the pre-trained layers) and another learning rate to the second layer group of fully connected layers.

I may be wrong, but I think that as you unfreeze the network, it get splitted, by default in three layer groups: the first half of the body, the last part of it, and the head. If you pass slice(a,b), you will get a applied to the 1st part of the body, b to the head, and something in the middle to the 2nd part of the body. Indeed, rather than slice(), you can try and pass a list like this: [a,c,b], a-la-fastai 0.7. It’ll work.

@adeperio

Is what I just said correct?

I think not, but let’s wait and see if more informed fellas will answer.

@adeperio

where it takes a max learning rate, and starts from a minimum value of max_lr/div_facto

Correct, but don’t confuse the “variation” of LR for the 1-cycle policy with differential learning rates. They are very different concepts, and as you train an unfrozen network, both of them are applied.
Suppose you start fit_one_cycle() upon an unfrozen net, just specifying a learning rate of lr=1, and a slice like (lr/a,lr). What follows will happen:

• It will do a cycle where the learning rate for the head varies between 1/25 and 1.
• The learning rate for the first group of the body will go between 1/25a and 1/a.
• The learning rate for the central group will vary accordingly within some middle ground. Looking at the code above, if layer_groups>3, the array will be spaced in proportion, unless you pass it explicitly.

@adeperio

If this is how fit_one_cycle operates, and it determines the minimum learning rate itself, how then do the learning rates in the slice passed to max_lr get used in each learning group by the 1cycle policy?

Read above, but ask more informed people for confirmation.

@adeperio

and doesn’t sound like nonsense.

Not at all!