Note: This is a wiki post - feel free to edit to add links from the lesson or other useful info.
Both sound great!
As we approach the end of part 1 are there any plans for a part 2? If so, any idea when?
Follow-up on this – any thoughts on what you’d like to cover in the next Part 2?
I will make a note about the part 2 question, and ask Jeremy at the break (since it is out of flow with the topic now, but I know people are wondering)
Also there might be wrong tagging by experts in the dataset which would have caused the model to get confused…
Extra Read: Here’s a Link to my interview with Leslie Smith (Author of Cyclical LR work).
Is the learning rate plot in lr_find plotted against one single mini-batch?
Why don’t we need minimum of the learning rate?
During the lr_find() method, every learning rate is applied to a different batch, right? Is the network reset to the initial status after each trial?
No, we take steps on 100 different mini-batches, not just one, increasing the learning rate at each one.
Here is the link to the paper Jeremy refers to.
+1 to this… is it the same minibatch everytime, or a different one? Are the weights updated each time?
There are other implementations of LR finder, including PyTorch Lightning and some community-written callbacks for Keras (ex: here)
Why would an “ideal” learning rate found with a single mini-batch at the start of training keep being a good learning rate even after several epochs and further loss reductions? Wouldn’t the ideal learning rate be a local property of the loss function?
For LR finder, why use the steepest and not the min?
No, since it’s not really training while the learning rate is too small, you don’t need that.
When should we run the learning rate finder? Only at the beginning or should we update after a couple of epoch?
There is a lr_min, lr_steep in the code which selects the minimum, maximum lr rate. Part of the lr_find function