Plotting loss comes empty after training

fastai
1

Hi folks,
While training the models, I encountered this problem frequently. The learn.recorder.plot_loss() returns an empty plot.
I looked at the source code and the recorder.losses is used to plot the losses, in my case the losses object is coming empty.
Has anyone ever seen this before and what could be the possible reason behind it?

2

Attaching the relevant images.

SCR-20220904-fj
SCR-20220904-fj1024×708 27.4 KB

and
SCR-20220904-5y
SCR-20220904-5y736×220 4.36 KB

3

Hey,
could you share more details? Like which data did you use, what kind of learner etc. Ideally throw together a notebook that produces this behavior. An uneducated (and maybe not very helpfull) guess would be that something with the recorder callback is messed up, since learn_8.recorder.losses is empty.

4

Thanks for the quick turnaround.
I am providing the link to the notebook in which you can see this behaviour.

The dataset was an open dataset of cloud images, 10 classes. I used many types of models it came in Convnext_tiny and convnext_small_in22k

5

Ahh, when you are initializing ClassificationInterpretation as well as when you are using learn_8.predict(…) the learner calls .get_preds which flushes the recorder. So using the recorder right after training, before you do anything else with the learner should solve your problem :slight_smile:

6

Unfortunately, the issue persists even on calling the recorder just after training. Still comes empty.
But I agree that somewhere, somehow the recorder is getting flushed out.

7

Thats weird. When I run .recorder.plot_loss() right after training it works for me every time.
You can check out this notebook. I copied yours and removed all but the parts that were relevant for learn_8, I also used learn_8.fit_one_cycle(1) to train faster, but everything else should be the same.

8

Hmm! Yeah, I can see that you are getting the plot if you plot before calling interpretation or predictions.
Let me check this again.
I will plot the loss just after training and will report in case I see an anomaly.

Thanks a lot for the help :slight_smile:

2 Likes
9

Several other functions, like learn.predict() or learn.show_results() will reset the recorder. However, I don’t like this behavior…

I didn’t yet found a easy way to prevent this. One would be to save all relevant attributes of the Recorder class like ‘lrs’,‘iters’,‘losses’,‘values’

1 Like