Getting Metrics on Validation Set

fastai
1

Hi

I am just getting started with FastAI. At the moment I am working on a small side project where I am planning to use Azure ML to deploy a CV model that can classify various fruits and vegetables from images. I created a model and trained it and all worked perfectly. I am now wondering how I can get certain metrics as plain numbers from my validation set (e.g. accuracy, auc/roc, F1 score etc.) for logging in Azure. My assumption was that these numbers can either be extracted from the “ClassificationInterpretation.from_learner(learn)” or from the learner itself. However, I could not manage how to get these general metrics. Is there some in-built functionality I missed? Do I need to calculate those metrics myself?

I spent all my morning searching for this in various places but could not find anything helpful. I would really appreciate some hints!

Thanks a lot in advance
Chris

1 Like
2

Hey Chris,

When training the model, you can specify which metrics you’d like to track on your validation set. For instance, in the text tutorial the learner is defined as follows:
learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)

While it’s training, the accuracy will be tracked. Hope this helps!

3

Hi @neural

That I know. I played around with it a little. What I am looking for, however, is a single accuracy number I can extract after the training. In other words, that would be the accuracy of the last epoch:
image
I would like to track the models training accuracy over multiple run with multiple parameters. I found how I can select various metrics but I didn’t find a way how to extract the metrics after the training to lets say a variable so I can use it afterwards in a training dashboard for example.

Is there some functionality like that?

4

Alright, I actually found a way to calculate the measures myself. Sharing here in case anybody else is ever looking for that.

I extracted the decoded and target values from the ClassificationInterpretation object as follows:

interp = ClassificationInterpretation.from_learner(learn)
y_hat, y = flatten_check(interp.decoded, interp.targs)

y_hat and y can then be used in the classic sklearn.metrics functions to calculate accuracy, F1 score etc.

from sklearn.metrics import accuracy_score, f1_score
accuracy = accuracy_score(y, y_hat)
f1 = f1_score(y, y_hat, average='weighted')
5

Just came across this question when trying to rebuild a model using v2 of the API.

Something like this can work

learn = cnn_learner(data, models.resnet34, metrics=accuracy)
learn.fit_one_cycle(4)
metrics=learn.recorder.metrics
print(metrics) # a list
print(metrics[0]) # AvgMetric
print(metrics[0].value) # Tensor
print(metrics[0].value.item()) # float

That will print something like this:

[<fastai.learner.AvgMetric object at 0x7fe87e64eb70>]
<fastai.learner.AvgMetric object at 0x7fe87e64eb70>
TensorBase(0.9966)
0.9965828657150269
3 Likes
6

Hi Jason: It Works!!

Wonderful tip: thank you!
How did you find out that :
print(metrics[0].value.item()) # float
Which means that metrics is a list, where the first value is the best value computed. It would be nice if you help me as well to understand how you managed to dig out this bit.
For now, THANKS A LOT!

Andrea

7

Andrea,

Wow, this is a blast from the past.

I am sure that I just drilled down by printing out the variables one by one until I got the answer that I was looking for.

Once I had metrics, I just did print(metrics), which output [<fastai.learner.AvgMetric object at 0x7fe87e64eb70>]. The square brackets indicate that it’s a list, but if you aren’t sure about an object’s type, you can use the built-in Python method type(), e.g., print(type(metrics))

Then I wanted to inspect one of the objects in the list, so I did print(metrics[0]), which output <fastai.learner.AvgMetric object at 0x7fe87e64eb70>.

I searched how to get the value from fastai.learner.AvgMetric, which is how I came up with print(metrics[0].value). That output TensorBase(0.9966).

I searched how to get a numeric value of a Tensor (or TensorBase), which is how I came up with print(metrics[0].value.item())

8

Jason,
Thanks a ton!
I will investigate t a little further so to familiarize myself with the procedure, but truly you solved a big unknown for me: how to dig out the values calculated in a way that I can directly put them in a dataframe, instead of having manually to go for it myself with the printout…

Thanks again!
Andrea

PS if someone would have spent half a day writing about this kind of procedures, we would not nee dto reverse-engineer a working library… which in the end limits its usefulness. I do not understand how the creators do not think in this way…

9

Hi Jason,

Unfortunately this is not exactly what I have been looking for. I simply would like the best(highest) vslu for the metrics during th calculation. Instead printing metrics[0].value.item()
outputs only the last value of the metrics. Like this:

Epoch Loss Val loss Metric Time
7 1.573036 0.800377 0.833333 00:02
8 1.433010 0.748553 0.866667 00:01
9 1.328349 0.715591 0.866667 00:01
10 1.225719 0.683847 0.833333 00:01
11 1.135028 0.661689 0.833333 00:02
12 1.071157 0.659765 0.833333 00:02

No improvement since epoch 8: early stopping
Value= 0.8333333134651184

So, it retain the last computed value for the metrics but not the highest. Do you know how to obtain as output the highest computed vale instead?