Val_loss changes depending on how you call loss_function

fastai
1

Hi everyone, i’m using fastaiV1 in a text classification problem. I have 2 classes (positive/negative). My training looks like this :

  "Total time: 07:24\n",
  "epoch  train_loss  valid_loss  accuracy\n",
  "1      0.657636    0.662783    0.747126  (00:16)\n",
  "2      0.643375    0.998169    0.754310  (00:18)\n",
  "3      0.633410    0.855394    0.748563  (00:19)\n",
  "4      0.619446    0.849962    0.797414  (00:18)\n",
  "5      0.615339    0.604372    0.795977  (00:15)\n",
  "6      0.614293    0.692293    0.794540  (00:16)\n",
  "7      0.601260    0.608897    0.784483  (00:18)\n",
  "8      0.611353    0.630321    0.814655  (00:17)\n",
  "9      0.596360    0.560087    0.790230  (00:17)\n",
  "10     0.623256    0.627829    0.788793  (00:16)\n",
  "11     0.591119    0.583458    0.814655  (00:17)\n",
  "12     0.594566    0.580568    0.794540  (00:17)\n",
  "13     0.617168    0.621369    0.808908  (00:16)\n",
  "14     0.606012    0.586289    0.797414  (00:17)\n",
  "15     0.593627    0.606417    0.806034  (00:17)\n",
  "16     0.592749    0.577795    0.806034  (00:16)\n",
  "17     0.590719    0.589672    0.831897  (00:18)\n",
  "18     0.595279    0.707443    0.814655  (00:18)\n",
  "19     0.571968    0.574797    0.830460  (00:17)\n",
  "20     0.572438    0.567295    0.816092  (00:17)\n",
  "21     0.588064    0.544141    0.814655  (00:19)\n",
  "22     0.579162    0.600066    0.821839  (00:19)\n",
  "23     0.571432    0.595173    0.817529  (00:16)\n",
  "24     0.577660    0.588137    0.823276  (00:19)\n",
  "25     0.578836    0.567601    0.817529  (00:19)\n",

It comes from the command :

 learn.fit_one_cycle(25, 0.0005, moms=_p.moms, wd=_p.wd)

Where learn is a RNNLearner.classifier. Its loss_func is :

<function torch.nn.functional.cross_entropy(input, target, weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='elementwise_mean')>"

However, when I do :

preds = learn.get_preds()
torch.nn.functional.cross_entropy(preds[0],preds[1])

my output is :

tensor(0.6243)

As you can see, the val_loss evaluated at my last step of training is 0.567601, while the loss i get from calling the same loss_func on the predictions on the validation set is 0.6243 (I have no test set in this example, only a validation set).

Any idea why those losses are different?

2

When you call learn.get_preds, you get the actual probabilities, which means your output went through a softmax. If you want the loss, you should then pass preds[0] through a log then apply F.nll_loss.

More easily, learn.get_preds(with_loss=True) will return probabilites, targets and losses.

1 Like
3

Thank you very much.