Why learn.lr_find() followed by learn.recorder.plot() gives different value each time when running even if I don’t run the learn.fit_one_cycle again? Please help.
Short answer. The data that you are passing is different for different runs due to various data augmentations used.
Below I show the process that I used to find this thing. (Most things were not necessary but the below process is a good way of debugging things).
So there are two functions are mostly responsible for this
learn.recoder.plot(). The first step is to make sure that the
state_dict of the model remains same. To do so I used the below code.
learn.model.state_dict() learn.lr_find() learn.model.state_dict() learn.recorder.plot() learn.model.state_dict()
The state_dict was same in all the cases. Now the next step is to check the internal state that we pass to
learn.recorder.plot(). The idea is if the internal states (more specifically the losses) are same, then the randomness is due to
learn.recorder.plot() else it is due to
The reason I use losses is
learn.recorder.plot() only manipulates the losses and there is no way to introduce randomness. I use the below code to check for the loss values.
learn.lr_find() learn.recorder.losses learn.lr_find() learn.recorder.losses
As it turns out the loss values were different in the above two cases. So now we know the randomness is introduced in the
learn.lr_find(). Next step is to check the source code of it. The source code is
start_lr = learn.lr_range(start_lr) start_lr = np.array(start_lr) if is_listy(start_lr) else start_lr end_lr = learn.lr_range(end_lr) end_lr = np.array(end_lr) if is_listy(end_lr) else end_lr cb = LRFinder(learn, start_lr, end_lr, num_it, stop_div) epochs = int(np.ceil(num_it/len(learn.data.train_dl))) learn.fit(epochs, start_lr, callbacks=[cb], wd=wd)
start_lr and end_lr are same for all the cases. So the problem must be in LRFinder. Now if we see the source code of LRFinder, we see the following lines.
class LRFinder(LearnerCallback): "Causes `learn` to go on a mock training from `start_lr` to `end_lr` for `num_it` iterations." def __init__(self, learn:Learner, start_lr:float=1e-7, end_lr:float=10, num_it:int=100, stop_div:bool=True): super().__init__(learn) self.data,self.stop_div = learn.data,stop_div self.sched = Scheduler((start_lr, end_lr), num_it, annealing_exp)
Here we found the problem.
self.data is not same for different iterations (due to various data augmentations used). Hence different loss values and thus different graphs.
If I run learn.lr_find() many times before learn.fit_one_cycle(). Will it affect my final result?
Also, will it affect accuracy of the model?
No it won’t, as it’s ‘mock’ training. So anything happening to those weights are wiped the moment lr_find() is done
save.data is not same in LRFinder for every iterations and gives different graphs. So, I’ll have different best learning rate(lr) points for every time I run
lr_find() command. Then which lr should I choose?
but i set no augmentation, i still got different plots, can u please help me out of the problem? Thank in advance!
Is there any follow up for this question ?