Lesson1: train_loss vs. valid_loss vs. error_rate

Hello Community,

i just finished lesson 1 and imported my own dataset.
I got a couple of questions to better interpret my results:

  1. Are train_loss and valid_loss percentages? If so what does it mean if they are above 1?
  2. What does it mean if train_loss and valid_loss stay high (above 1 or 2) but the error rate or accuracy gets pretty good (below 0.05)?
  3. What does it mean when the train_loss is reducing nicely near to 0 but the valid_loss is stuck on above 1?
  4. In python notebook if I do e.g. learn.fit_one_cycle(8) and follow up by an other learn.fit_one_cycle(5) is that the same as doing an initial learn.fit_one_cycle(13) ?

Thank you for the help

No, they are the values on your loss function (found in learn.loss_func), which is normally CrossEntropy.

It’s moreso if the train and validation loss are dropping while your metric (here it is accuracy) is either increasing or decreasing (whatever is good)

You’re overfitting

No, it is not. fit_one_cycle has some special paramters that it calls during fit, specifically it warms up the optimizer and reduces it at the end (your learning rates), and it’s based on that one call. So 2 calls != 1 longer call

4 Likes

Thank you @muellerzr

One more question regarding interpretation:
My model classifies between “Herrenschuhe” (shoes for men) and “Damenschuhe” (shoes for women).
The interp.plot_top_losses function yields the following output:

It has an overall accuracy of 97% (0.97).

Please tell me if i am correct, I interpret the above picture in this matter:

The first shoe was predicted as a “Herrenschuh” but actually is a “Damenschuh”. It has the highest loss within my model of 5.93. The loss is so high that the corresponding probability is 0. At this point the model just took a guess and predicted a “Herrenschuh” but lost the coin flip in this case.

The second shoe was predicted as a “Damenschuh” but actually is a “Herrenschuh”. It has a high loss of 2.55 which corresponds to a probability of 8%. The model is 8% certain that its prediction is “good”.

The third shoe was predicted as a “Herrenschuh” and actually is a “Herrenschuh”. It has the third highest loss of 0.45 in my dataset, that is why its listed even though it was predicted correctly. A loss of 0.45 corresponds to a probability of 64%. The model was 64% certain that its prediction is good. Same thing for the second row.

Please tell me if i am correct or not and why.
Thank you

Yes, you are right.