How to get per label loss with get_preds?

fastai fastai dev
1

I used to be able to do this to get loss per label for my multi-label tasks:

probs, targs, loss = learn.get_preds(DatasetType.Valid, with_loss=True)

print(f'Validation Loss: {loss.mean()}')
print(f'Validation Loss (per label): {loss.mean(dim=0)}')

For example, if I’m predicting 8 labels … loss comes back as a (bs,8) tensor.

Now it appears that loss is returned for each example in v2 (loss comes back as a (bs,) tensor):

probs, targs, loss = learn.get_preds(dl=dls.valid, with_loss=True)

print(f'Validation Loss: {loss.mean()}')
print(f'Validation Loss (per label): {loss.mean(dim=0)}')

… just returns the avg. loss.

Is there any way to get that behavior with v2?

2

I would say it was a bug of v1 that we fixed in v2. For your usage, you can either"

  • not call with_loss=True then compute those losses directly at the end (just keep in mind preds went through the activation function)
  • or add a callback to your Learner to save those losses.
1 Like
3

A fortuitous bug I guess :slight_smile:

A side-related question, is there any way to pass in a threshold to learn.predict()? It would be helpful for multi-classification tasks … but most helpful for multi-label tasks where the threshold is often much less that .5.

Thanks

4

The threshold is in the loss function, so you can adjust at any time you like (learn.loss_func.thresh if my memory serves me right).

5

Yes that’s it. Thanks!

As the optimal threshold is something I have setup as a metric … is there any easy way to grab the final value of that metric after training? Something like learn.metrics['opt_th'] (opt_th being the name of my metric)?

For example:

def get_metric(metric_name:str):
    return self.metrics.filter(lambda el: el.name == metric_name.strip())[0]

Also, is there a way to update the loss function’s threshold in a metric before other metrics are calculated?

… and btw, I’m open to submitting PRs … I just don’t want to mess with you guys as ya’ll are in the middle of finishing up v2 and with the upcoming course/book. just lmk.

6

Mmm, sounds like you would better have this in a Callback than a metric. Any callback sets itself as an attribute of learner with a snake-thing name, so you could then access your threshold (and set it to the loss function) with learn.best_thresh.thresh (if the callback is called BestThresh). If you run it before the Recorder (with `run_before=Recorder) then it should work I guess?

7

Yah that makes sense … like in v1.

run_before=Recorder is nice to know about.

8

Is there a way to add the result of this callback (e.g., the optimal threshold) as a column to the table that displays during training alongside items like epoch, train_loss, valid_loss, <metrics…>, time?

9

Not if there is no metric associated to it AFAICT. Have a look at the superres lesson port to v2, there are several things plotted by I don’t remember how it was done.

1 Like
10

It’s not in there…

<ipython-input-57-730be686d513> in setup(self)
     33         # one time setup code here.
---> 34         self.opt_thresh = self.loss_func.thresh
     35         self.learn.metrics = self.learn.metrics + L(ValueMetric(self.metric_value, 'opt_th'))

AttributeError: 'BaseLoss' object has no attribute 'thresh'

Any other ideas where this might set (it’s gotta be somewhere for learn.predict :slight_smile: )

11

If it’s not there, it means you are using a loos function that does not have a threshold. It is in BCEWithLogitsFlat as you can see here.

1 Like
12

ah that would make sense.

I’m trying to figure out how to use synth_learner for creating tests for my callback and it’s not a multi-label friendly one by default. Asked in another post, is there a way to return a synth_learner that is set up as a multilabel learner?

13

You will have to build a new one for that.

14

Or … maybe a different synth_dbunch?