in course_v3, lesson 1, how to make an early stopping for learn.fit_one_cycle when the error_rate is low enough or just save the model of lowest error_rate ?
because i really dont know how many cycles( the parameter) i should use in the fit_one_cycle function… some times the epochs used is too many or too small that have missed the best one.
Do i need to write a callback function by myself ? How to implement that…
June 19, 2019, 4:52pm
Because of the way fit-one-cycle works, I’m not sure if early stopping would be as effective.
I have a callback that save the last best model, and from practice it’s frequently the last epoch
callbacks = [SaveModelCallback(learn, every='improvement', monitor='val_loss', name=save_name)]
learn.fit_one_cycle(epochs, lrs, pct_start=pct_start, callbacks=callbacks)
Thanks! i have found SaveModelCallback that you mentioned
here, the offical example was used in fit_one_cycle, so it should work…emm, i will try
December 21, 2019, 8:58pm
I tried the SaveModelCallback(), it does save progressed pth files, but it does not stop training even when criteria are met.
Do we have some real early stopping callbacks?
December 21, 2019, 9:05pm
What do you mean by “when the criteria is met?” How are you setting it up?
December 22, 2019, 6:17am
@muellerzr thanks for asking.
I set up monitoring on error_rate for each epoch for model= min, also tried on each improvement
Please let me know if I was missing anything.
December 22, 2019, 6:17pm
For reference, the SaveModelCallback will only save out the ‘best’ model, it will not stop the training cycle.
i.e. if you setup and run 20 epochs, and the best model is at epoch 17, training will continue until the full 20 regardless.
It sounds like you are looking for something that stops training when a criterion (accuracy, etc) is met but I’m not aware of anything like that already written - you’d have to setup a custom callback for it I think.
Hope that helps!
December 22, 2019, 6:40pm
IIRC there’s an EarlyStoppingCallback
December 22, 2019, 7:14pm
@muellerzr - didn’t realize it existed.
I see it noted in the docs now but with no usage example:
@franva - the above is what you’ll want to use then.
I’ve been meaning to add some improvements to SaveModelCallback so if I get time I’ll post those out for review when done and maybe see if I can make an example for EarlyStoppingCallback to improve the docs as well.
December 22, 2019, 7:16pm
December 22, 2019, 7:51pm
thanks for the link
That is odd how it’s laid out (one without example that shows higher in search, another buried but with example)…but glad there is an actual example out there.
December 23, 2019, 5:57am
@muellerzr and @LessW2020,
Thank you all for your help
The forums and especially the people here are very helpful~!!!
I will give it a try once am back to home.
Fast.Ai the best AI framework I have found so far~!