Not entirely sure what’s going on with this but there appears to be a bug in the EarlyStoppingCallback code as shown here: https://docs.fast.ai/callbacks.tracker.html#EarlyStoppingCallback
The error output in the documentation is the same error you get running the code in a notebook so I’m assuming the error has been included in the documentation when it was generated automatically?
The error seems to disappear when the metric is switched from ‘accuracy’ to ‘mean_squared_error’, though I have not tested for other metrics.
Also after changing the metric, the callback does not trigger after the patience period even when the condition has been met (i.e. doing 10 more update steps with no improvement when patience is set to 5)
Something worth noting is that I am using Windows which I realize is unsupported. However as this error appears in the official documentation I expect the error runs slightly deeper than Windows compatibility.
I can try to put together a minimum reproducible example of the behavior error (not stopping) that does not use confidential data when I’m free later.
I will of course try to reproduce this on Linux as I’m sure this is more helpful to the dev team.
I don’t have time right now to dig into the code for the root cause, but I am happy to do this if it hasn’t been addressed in a day or so.