Add MAPK metric to fast-ai library

Part 1 (2019)
1

Hi all,

Kaggle has some competitions that use MAPK as evaluation metric(whale identification,doodle recognition…etc), but currently fast-ai metric only supports top-k accuracy. I am wondering if a PR to add MAPK is needed?

According to this poston Kaggle, I implemented the MAPK and tested on fast-ai Cifar10 dataset.

The idea is simple,

  1. calculate single_prediction MAPK value
    If targs has batch size (m, classes), then single_pred has size (1,classes)

  2. take average of the single_prediction MAPK of the batch, fast-ai library will take care of the whole validation set.

def mapk(preds,targs,k=5):
batch_pred = preds.sort(descending=True)[1] #batch_size * classes
return torch.tensor(np.mean([single_map(p,l,k) for l,p in zip(targs,batch_pred)])) #return tensor instead of npdarray

def single_map(pred,label,k=5):
try:
return 1/ ((pred[:k] == label).nonzero().item()+1) #scalar division
except ValueError:
return 0.0

Here are some test result:

By default, it calculates MAP5
MAP5

You can also use partial(mapk,k) to change it to desired MAPk metric.
Here is an example of MAP1, it should match accuracy since it is calculating MAP1

MAP1
MAP1.JPG939×359 22.1 KB

2

Hi,

I used the same map@k function but getting the following error while training:

in mapk(targs, preds, k)
1 def mapk(targs,preds,k=5):
2 batch_pred = preds.sort(descending=True)[1] #batch_size * classes
----> 3 return torch.tensor(np.mean([single_map(l,p,k) for l,p in zip(targs,batch_pred)])) #return tensor instead of npdarray
4
5 def single_map(label,pred,k=5):

in (.0)
1 def mapk(targs,preds,k=5):
2 batch_pred = preds.sort(descending=True)[1] #batch_size * classes
----> 3 return torch.tensor(np.mean([single_map(l,p,k) for l,p in zip(targs,batch_pred)])) #return tensor instead of npdarray
4
5 def single_map(label,pred,k=5):

in single_map(label, pred, k)
5 def single_map(label,pred,k=5):
6 try:
----> 7 return 1/ ((pred[:k] == label).nonzero().item()+1) #scalar division
8 except ValueError:
9 return 0.0

IndexError: dimension specified as 0 but tensor has no dimensions