Accuracy_multi

Part 1 (2020)
1

For accuracy_multi we apply the sigmoid to our activations (to make them between 0 and 1), we need to decide which ones are 0s and which ones are 1s by picking a threshold. Each value above the threshold will be considered as a 1, and each value lower than the threshold will be considered a 0:

def accuracy_multi(inp, targ, thresh=0.5, sigmoid=True):
    "Compute accuracy when `inp` and `targ` are the same size."
    if sigmoid: inp = inp.sigmoid()
    return ((inp>thresh)==targ.bool()).float().mean()

^^^From lectures, what i dont understand is where in this code does it say to only compute
accuracy if inp and targ are the same size ? Also what if they aren’t the same size
i.e Model recognises 1 image while target has 2 predictions by not calculating accuracy on this aren’t we losing valuable data ?

2

Hi @shay1309, I wrote a little Colab notebook explaining what it means by “when inp and targ are the same size.” It basically means that the targ tensor should be one-hot encoded. It is assumed that inp and targ will both have the same number of examples.

1 Like
3

Hi @GoofyMango just had a look at your collab, the explanations are really clear appreciate it alot but I have 1 qn which is : What if the target has an image of a chair and an aeroplane but our predicition only get’s labelled as having a picture of a chair ? Does that mean our accuracy is absolutely 0 since we don’t even consider this prediction?

4

Hi @shay1309, I added a new section to the notebook to address your question, but it’s basically that, in the multi-class classification case, we consider each class separately when calculating accuracy.

Feel free to copy the notebook too and try out experiments on any questions you have, I find that really helps me understand things more deeply.

2 Likes
5

@GoofyMango Just went through your notebook. I love it. Thanks

1 Like
7

Hey man just had a look at ur notebook ,really appreciate the effort man, it helped loads.

1 Like
8

Also man , Say i go into the "real world " and i have a new csv file of target values and a new folder with images how do I test model to predict and compare with the csv ? I could iterate through and compare but I’d think fast ai has something for that … would you happen to know?

9

Building up on the possible resources: I made a blog on accuracy_multi that discusses whether it is measuring the accuracy for each item, versus for each image. I saw that it was measuring the accuracy per item.

I also elaborated on the individual steps of the function to help beginners like me understand the underlying concept.

See the blog here:

Maria

2 Likes