CosAnneal#calc_lr in the if statement we are first making the learning rate very small for first 20 batches?
def __init__(self, layer_opt, nb, on_cycle_end=None, cycle_mult=1):
self.nb,self.on_cycle_end,self.cycle_mult = nb,on_cycle_end,cycle_mult
def calc_lr(self, init_lrs):
self.cycle_iter += 1
This is the tiny flat line on this graph starting at iterations = 0?
This is the good ol’ trick from part 1 v1 of first training with super small lr to get out of finding easy but sucky optima and then only increasing the learning rate to what we would like to use for training?
Just wanted to confirm I am reading this right and not going crazy Though not sure if reading this right actually precludes the second part of the statement from being true