Validating a model in image classification

For image classification,

  • What is the correct way to test a model against a labeled test set and collect the results in something like a confusion matrix? Are there examples, for the current fastai, that tells us what to do?. Note that I only have the model as exported with learn.export(), and I know the model’s input are to be of some size, and normalized according to imagenet stats.

  • And what is the difference between learn.get_preds() and learn.validate()? I’m confused as to which to use to check a model’s accuracy on a labeled test set as I tried them both but get different results on a test set. For example I do:
    item_tfms = Resize(128, ResizeMethod.Squish) batch_tfms = Normalize.from_stats(*imagenet_stats) test_dls = ImageDataLoaders.from_folder(Path(r"C:\Users\iai\Documents\SLACA_data\test"), train = "test_convert", valid = "test_convert", item_tfms = item_tfms, batch_tfms = batch_tfms, num_workers = 0) action_learn0.dls.valid = test_dls.valid

    Then I calculate accuracy using get_preds and validate as follows:

    probs, y_grnd, pred_class, loss = action_learn0.get_preds(dl = action_learn0.dls.valid, with_loss= True, with_decoded = True)

    acc = accuracy(probs, y_grnd)
    print(acc)
    acc2 = np.mean(pred_class.numpy() == y_grnd.numpy())
    print(acc2)

    acc and acc2 are the same.

    But when I do: action_learn0.validate(), the loss is the same but the accuracy validate()[1] is a few points less than acc.
    So which one is best to use and what are the expected outputs?

  • And how do we prepare the test dataset? Do we create a ImageDataLoader? If so, how do we keep the loader from forcing a split? Or do we use the test_dl function of learner.dls class?

Fastai can generate confusion matrices using ClassificationInterpretation (see here and check source code).

This post might be helpful regarding creating a test dataloader and using it to get confusion matrices:

Thanks for the information. The post is very helpful