Test set evaluation - how?

Part 1 (2019)
1

Hi everyone

Sorry if this is obvious - I have a train and test set sitting in two pandas dataframes. How do I rerun the learner on the test set? I tried:

test_data = ImageDataBunch.from_df('', test, ds_tfms=tfms, size=224, label_col=2)
learn.data = test_data
learn.validate()

but the results don’t make sense to me.

Regards
Ron

2

Once you have trained learn, you can use learn.get_preds() to get the validation

This will give you a list with 2 tensors inside it. The first tensor is the predictions and the second is the true values.

If you are trying to get predictions from test, you can instead use learn.get_preds(is_test=True)

This will give you the predictions in the first tensor and all 0s in the second tensor.

Assuming you used this line:

preds = learn.get_preds(is_test=True)

preds[0] would give you the output of something like this:

tensor([[ 2.6747, -1.6915, -2.7530,  ..., -0.0572,  1.0603,  2.3969],
        [ 0.2564, -0.1261,  1.7345,  ...,  2.5312,  1.3247,  3.9575],
        [ 0.0725, -1.2854,  5.8897,  ...,  1.8020,  4.0222,  0.0308],
        ...,
        [ 2.7146, -0.2381, -1.7115,  ..., -1.4560, -0.7291,  3.7035],
        [ 0.6296, -1.2794,  1.8849,  ...,  3.3482,  2.1478,  2.8770],
        [ 3.9928,  0.3684,  2.3774,  ...,  3.2413,  1.2406,  3.8148]])

Each row of this would be a prediction for each class so you want to know which number is the highest for each row. You can do this with argmax like so:

class_guess = preds[0].argmax(dim=1)

The other thing is I think you may be using ImageDataBunch.from_df a bit incorrectly. Instead of using it like that, you should add the test to the earlier databunch. If for some reason you had to do things like this, you would need to pull the train_dl out of test_data (since that is where you put your test set) and feed that into the learn.validate function so it would look like this learn.validate(test_data.train_dl) I’m not quite sure what that learn.data = test_data is doing so maybe it is doing what you want it to.

Edit: some of what I said earlier may have been incorrect about how you are using the test_df. It looks like from_df has a test available to pass, but it is looking for a folder of images, not a dataframe. I’d be curious to see what your test dataframe looks like, maybe I could help implement that if test is a dataframe instead of a Path.

Edit2: I’m re-evaluating my knowledge of learn.validate because it didn’t do what I expected it to. Basically everything I told you is wrong so back to the drawing board for me. I think this will work potentially (still testing though)

test_results = get_preds(learn.model, test_data.train_dl)
6 Likes
3

I had a problem with this also, using ‘from_folder’.

data=ImageDataBunch.from_folder(train=path_train,valid=path_val, path=path, test=path_test, ds_tfms=get_transforms(),size=224)

I was using the Chest X-Ray Pneumonia binary classification dataset from Kaggle (https://www.kaggle.com/paultimothymooney/chest-xray-pneumonials), which contains the test set in separately labelled folders (same as training & validation sets). When I try to get_preds(is_test=True) for the test set, I get a NoneType error. I glanced over the FastAI documentation and its seems that the test set has to be in a single folder? I.e. Not two separately labelled folders for this binary classification problem?

Thought was worth pointing this out if anyone stumbles on this topic and is using similar data / from_folder loading method.

Relevant documentation: https://docs.fast.ai/vision.data.html#ImageDataBunch.from_folder

Does this mean that we have to use a single unlabelled folder for the test set and then write our own function to measure accuracy etc? Or is it best just to create another data object for this data only and set it to validation again? I.e. you can just create a test_data object as above setting the path_test to be the valid(ation) set.

4

Hi @Thalantyr. Did you figure this out? I’m also using the X-ray pneumonia data set to learn fast.ai and cannot figure out how to evaluate the test set.

5

See the discussion here:

6

Thank you. How do I apply images to the same framework that pulls from a data frame? I have a test folder consisting of two folders for each classification in a binary classification problem.