Evaluate trained model on labelled test set

fastai
1

I understand how Fast AI define test set. Test set is meant to be unlabelled. However, I have a trained a model with ResNet18 and split my train set into 80:20. Now, I have the model trained, and wanted to evaluate the model with completely new, unseen data. How can I achieve that? All the topics similar to this question are from 2018-2019 and some of the function is not available in v2.

What I came up with is as follows:

import os
    
for i in range(10):
      v,s=0,0
      for filename in os.listdir('/content/AUC-DDD-v2/v2/test/c{}'.format(str(i))):
          if filename.endswith(".jpg"):
              probs, _ = learn.get_preds(dl=learn.dls.test_dl(filename, drop_last=False))
              if i == torch.argmax(probs).item:
                s+=1
              else:
                v+=1
                s+=1
      print('class {}: {}'.format(i,(s-v)/s))

My question is, is there an elegant way to solve this instead of using the function above? Thank you.

2

I’m new to this, but wouldn’t you just export your model, this creates a pickle file of your model, and then open that pickle file for use, such as:

#train your model
learn.export()
.
.
.
#use your model
path = 'C:\HOME\.fastai\data\...\your_model.pkl'
learn = load_learner(path)
is_it_whatever,_,probs = learn.predict(whatever_is_being_checked)

I did this with the “cat” code in 01_intro and it works when I check other pictures with the provided code.

1 Like
3

Do:
fnames = get_files(path/“test”, extensions=“.jpg”)
dl = learn.dls.test_dl(fnames)
learn.get_preds(dl=dl)

path in this case is Path(“AUC-DDD-v2/v2/test”)

If you specifically want the c, do get_files iteratively over each folder and add them together to merge them.

Predict is the wrong approach here as you’re doing each image individually, and following the other answer, you’re doing it on the CPU as well. The first was slower, the second is even slower.

2 Likes
4

Thanks for the reply, is it possible to couple the test result to ClassificationInterpretation so i can view the conf matrix too?

Also, if i set my unseen test data as valid dataset for learner, the data will not be considered while learning the model, am I true?

5

I tried to write my own code, not so optimize, but serve the purpose.

def getAccByClass(classes,cls):
  fnames = get_files(path/classes, extensions='.jpg')
  dl = learn.dls.test_dl(fnames)
  a=learn.get_preds(dl=dl)
  corr=0
  for i in a[0]:
    values, indices = torch.max(i, 0)
    if indices.item() ==cls:
      corr+=1
  print('class:{}, acc: {}, correct: {}, total:{}'.format(classes,corr/len(a[0]),corr,len(a[0])))
  return corr,len(a[0])
accuracy,total=0,0
for i in range(10):
  acc,leng=getAccByClass('c{}'.format(str(i)),i)
  accuracy+=acc
  total+=leng
print('total accuracy: {}'.format(accuracy/total))