How to deploy the model on a test set with subfolders as labels

Part 1 (2019)
1

Hi everyone,

I have built a model and validate it in the validation set. Now I want to check its performance on the test set (I have 10 subfolders each subfolder name means the label for that group of images) How can I do that and compute the accuracy for it?

Note: I have my test set folder created in the data bunch already as in the following code:

data = (ImageItemList.from_folder(path)
.split_by_folder()
.label_from_folder()
.add_test_folder(’./test’)
.transform(tfms=no_augumentation, size=224)
.databunch(bs=bs)).normalize(imagenet_stats)

Your help would be appreciated.

2

By default, you can’t do this, because fastai test-set doesn’t take any label (even if you pass label to it)

Once you done training, you can supply the test set as validation set

You can refer to this the fastai doc datablock API part, it has a detailed example :slight_smile:

https://docs.fast.ai/data_block.html#LabelLists.add_test_folder

3

I believe you will have to pre-process your data a little. Add all the images in a folder called test. Add this test set to your data bunch like you have already done. Then you can run the following line of code to make predictions on it:
preds,_ = learn.get_preds(ds_type=DatasetType.Test)
Use these predictions to calculate the accuracy. For this maybe you can create a csv with image names, actual targets and predicted targets. Work around a little, write short scripts for doing these things and you will be fine.

1 Like
4

thanks @heye0507 for the suggestion I tried to do that but when it comes to show the confusion matrix it returns the one belong to the real validation set not the test set.

5

Thanks @dipam7 for your help, I also tried this but I couldn’t compare the predicted with the actual label!
can you please help me with it?

and is it possible to show the top losses and confusion matrix of the test set?

6

Did you create another interpreter?

interp = ClassificationInterpretation(learn, preds, y, losses)
#or interp = from_learner(learn)
interp.plot_confusion_matrix()

#or interp.plot_top_losses()

see if that works :grinning: