How to make early stopping in fastai

Part 1 (2019)
1

in course_v3, lesson 1, how to make an early stopping for learn.fit_one_cycle when the error_rate is low enough or just save the model of lowest error_rate ?

because i really dont know how many cycles( the parameter) i should use in the fit_one_cycle function… some times the epochs used is too many or too small that have missed the best one.

Do i need to write a callback function by myself ? How to implement that…

2 Likes
2

Because of the way fit-one-cycle works, I’m not sure if early stopping would be as effective.

I have a callback that save the last best model, and from practice it’s frequently the last epoch

callbacks = [SaveModelCallback(learn, every='improvement', monitor='val_loss', name=save_name)]
learn.fit_one_cycle(epochs, lrs, pct_start=pct_start, callbacks=callbacks)
4 Likes
3

Thanks! i have found SaveModelCallback that you mentioned here, the offical example was used in fit_one_cycle, so it should work…emm, i will try
:grinning:

4 Likes
4

hi there,

I tried the SaveModelCallback(), it does save progressed pth files, but it does not stop training even when criteria are met.

Do we have some real early stopping callbacks?

5

What do you mean by “when the criteria is met?” How are you setting it up? :slight_smile:

6

Hi @muellerzr thanks for asking.

I set up monitoring on error_rate for each epoch for model= min, also tried on each improvement

Please let me know if I was missing anything.

Thanks

7

For reference, the SaveModelCallback will only save out the ‘best’ model, it will not stop the training cycle.
i.e. if you setup and run 20 epochs, and the best model is at epoch 17, training will continue until the full 20 regardless.

It sounds like you are looking for something that stops training when a criterion (accuracy, etc) is met but I’m not aware of anything like that already written - you’d have to setup a custom callback for it I think.

Hope that helps!

1 Like
8

IIRC there’s an EarlyStoppingCallback :wink:

1 Like
9

Thanks @muellerzr - didn’t realize it existed.

I see it noted in the docs now but with no usage example:

https://docs.fast.ai/callbacks.html#EarlyStoppingCallback

@franva - the above is what you’ll want to use then.

I’ve been meaning to add some improvements to SaveModelCallback so if I get time I’ll post those out for review when done and maybe see if I can make an example for EarlyStoppingCallback to improve the docs as well.

2 Likes
10

@LessW2020 the docs are weird for it but here’s an example a bit buried

https://docs.fast.ai/callbacks.tracker.html#EarlyStoppingCallback

3 Likes
11

thanks for the link @muellerzr.
That is odd how it’s laid out (one without example that shows higher in search, another buried but with example)…but glad there is an actual example out there.

12

Hi @muellerzr and @LessW2020,

Thank you all for your help :slight_smile:
The forums and especially the people here are very helpful~!!!

I will give it a try once am back to home.

Fast.Ai the best AI framework I have found so far~!

3 Likes