For image classification,
-
What is the correct way to test a model against a labeled test set and collect the results in something like a confusion matrix? Are there examples, for the current fastai, that tells us what to do?. Note that I only have the model as exported with learn.export(), and I know the model’s input are to be of some size, and normalized according to imagenet stats.
-
And what is the difference between learn.get_preds() and learn.validate()? I’m confused as to which to use to check a model’s accuracy on a labeled test set as I tried them both but get different results on a test set. For example I do:
item_tfms = Resize(128, ResizeMethod.Squish) batch_tfms = Normalize.from_stats(*imagenet_stats) test_dls = ImageDataLoaders.from_folder(Path(r"C:\Users\iai\Documents\SLACA_data\test"), train = "test_convert", valid = "test_convert", item_tfms = item_tfms, batch_tfms = batch_tfms, num_workers = 0) action_learn0.dls.valid = test_dls.validThen I calculate accuracy using get_preds and validate as follows:
probs, y_grnd, pred_class, loss = action_learn0.get_preds(dl = action_learn0.dls.valid, with_loss= True, with_decoded = True)
acc = accuracy(probs, y_grnd)
print(acc)
acc2 = np.mean(pred_class.numpy() == y_grnd.numpy())
print(acc2)acc and acc2 are the same.
But when I do:
action_learn0.validate(), the loss is the same but the accuracy validate()[1] is a few points less than acc.
So which one is best to use and what are the expected outputs? -
And how do we prepare the test dataset? Do we create a ImageDataLoader? If so, how do we keep the loader from forcing a split? Or do we use the test_dl function of learner.dls class?