Dear All,
There is some observation i had upon seeing validation loss reduction trend for the models that i have trained while working kaggle problems so far.
- At high LR (initial one found using find lR at training start) difference between Tr Loss and Val Loss is less
2)Upon training the model for more epochs if suppose we are also using ReduceLR on plateau call back then it would reduce the LR for faster reduction of Tr Loss if Tr loss dsnt improves
- After reduction of LR though TR loss start getting to reduce faster as before but gap between Val loss and Tr Loss starts widening which could be indicative of fact that we are over-fitting Training Set . Actually I do see its effect in the form of reduction of Kaggle score on Private LB /Public LB if i get to achieve very low Tr loss and loss Val loss but with Widened gap between two.
If my above observation were to make sense shouldnt it be a good idea to make CallBack ReduceLR on plateau bit more intelligent in terms of defining a criteria for reduce the LR (Currently if it sees less improvement in Tr Loss ,it reduces LR by certain pct,please correct me if not true )
say: As in it starts seeing loss gaps widening then it should step up the LR at the end of epoch so in next LR cycle higher value should be used so we get more generalized model ?