%99 Accuracy on Valid Data, %1 Accuracy on Test Data, What Am I Missing?

keremnymn · October 13, 2020, 1:33pm

Hello,
For my project, I need to train a model that can recognize handwritten “D, T, F, Y” letters. So I downloaded this dataset. And then I extracted the letters I need, and turn them into .png files using this script.

And then first of all I inverted their colors and added black border:

from PIL import Image, ImageOps

from imutils import paths

paths = paths = list(paths.list_images(’/content/content/abece’))

for i in paths:

img = Image.open(i)

inverted_image = PIL.ImageOps.invert(img)

img_with_border = ImageOps.expand(inverted_image,border=4,fill=‘black’)

img_with_border.save(i)

And then added a white border:

for i in paths:

  img = Image.open(i)

  img_with_border = ImageOps.expand(img,border=4,fill='white')

  img_with_border.save(i)

so they look like this:

So step 1 and 2 are for making the data similar to expected inputs.

Then, I built the data:

np.random.seed(42)
data = ImageDataBunch.from_folder(path, train="abece", valid_pct=0.20, num_workers=4).normalize(imagenet_stats)

Model trained for %99 accuracy with resnet18 pretrained model.

matrix

I export the model:

learn.export()

And run the model on the test data that it’s never seen before:

from fastai.vision import learner, load_learner, open_image

from IPython.display import Image, display

learn = load_learner("/content/content/","export.pkl")

from imutils import paths

paths = list(paths.list_images('/content/test'))

for resim in paths:

  model = learn.model

  model = model.cuda()

  img = open_image(resim, div=True, convert_mode='L')

  prediction = learn.predict(img)

  display(Image(filename=resim))

  print(prediction[0])

  print(resim)

Results:

I tried converting the data to JPG, or converting the test data to both JPG and PNG, no luck. It just doesn’t understand the letters I’m putting. There must be something I’m missing important here.

sut · October 13, 2020, 2:31pm

Due to the above, your train/valid are being normalized but your test set is not. You can use the .test_dl() functionality to mimic the pre-processing steps (like normalize) being applied in valid to test.

See:

muellerzr · October 13, 2020, 2:35pm

He’s using predict though which builds a test_dl anyways

That being said, it would make more sense to try the test_dl method and get predictions with get_preds rather than running predict repeatedly. I can’t inherently see what could be going wrong here. What I would recommend is predict on one image by doing learn.predict() and pass in the filename to see what it does

sut · October 13, 2020, 2:47pm

Good point Zach. I’d start by making sure that you could replicate the 99% accuracy on the train and valid in your prediction loop. So change the line to:

paths = list(paths.list_images('/content/valid'))  #instead of content/test/
for resim in paths:

keremnymn · October 13, 2020, 3:47pm

Thank you very much for your answers.
I get these errors, can you help me get back on the track:

I changed dataloader to this:

data = ImageDataBunch.from_folder(path, train=“abece”, valid_pct=0.20, test="/content/test", num_workers=4).normalize(imagenet_stats)

print(data.test_dl):

DeviceDataLoader(dl=<torch.utils.data.dataloader.DataLoader object at 0x7fa46b73a5f8>, device=device(type=‘cuda’, index=0), tfms=[functools.partial(<function _normalize_batch at 0x7fa495603a60>, mean=tensor([0.4850, 0.4560, 0.4060]), std=tensor([0.2290, 0.2240, 0.2250]), do_x=True, do_y=False)], collate_fn=<function data_collate at 0x7fa497f9dd90>)

I tried to get predictions:

learn = load_learner("/content/content/", test=data.test_ds)

preds,y = learn.get_preds(ds_type=DatasetType.Test)

print(y)

the error I’m getting:

/usr/local/lib/python3.6/dist-packages/PIL/Image.py in open(fp, mode)
2813 fp.seek(0)
2814 except (AttributeError, io.UnsupportedOperation):
-> 2815 fp = io.BytesIO(fp.read())
2816 exclusive_fp = True
2817

AttributeError: ‘numpy.ndarray’ object has no attribute ‘read’

If I change the code in 3rd step to this:

learn = load_learner("/content/content/", test=ImageList.from_folder("/content/test"))

preds,y = learn.get_preds(ds_type=DatasetType.Test)
print(y)
print(len(y))
print(len(os.listdir("/content/test")))

output:

tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0])
32
33

actual classes:

print(data.classes)

output:

[‘D’, ‘F’, ‘T’, ‘Y’]

if I understood correctly, when I run get_preds the model thinks all the Fs in the test folder are Ds which is completely wrong if I’m not skipping anything.

But the problem is that I didn’t normalize the data at the last step which gave me zeros as predictions.

So another problem arises: how to normalize the data in test folder and run get_preds on it?

UPDATE:
I think I figured out how to get test data from databuilder:

predictions, *_ = learn.get_preds(DatasetType.Test)

labels = np.argmax(predictions, 1)

print(labels)

output:

tensor([0, 0, 2, 1, 2, 2, 1, 1, 0, 0, 2, 0, 2, 0, 2, 2, 0, 2, 2, 2, 2, 2, 2, 0,
2, 2, 1, 1, 1, 0, 2, 2])

which are the same predictions in the first post. I think these are normalized predictions, am I right?

keremnymn · October 13, 2020, 3:49pm

This is actually pretty reasonable, but since I use valid_pct in databuilder, I don’t know where my valid folder is. But I’ve got top losses:

UPDATE:
I’ve found a way to get predictions from valid_pct and there seems to be no problems:

for resim in data.valid_ds.items[:10]:

model = learn.model

model = model.cuda()

img = open_image(resim)

prediction = learn.predict(img)

ad = os.path.basename(resim)

ad2 = os.path.join(’/content/content/abece/’, ad[8], ad)

display(Image(filename=ad2))

print(prediction[0])

print(resim)

output:

ANOTHER UPDATE:

It may be indeed a normalization problem:

from matplotlib import image

from matplotlib import pyplot

trained_image = image.imread(’/content/content/abece/D/cikar_D_D-8231.jpg’)
test_image = image.imread(’/content/content/test/test1.jpg’)

data = asarray(test_image)
data1 = asarray(trained_image)

print(data)
print("\n")
print(data1)

output:

[[249 253 253 244 … 241 238 238 239]
[229 242 252 251 … 243 241 239 239]
[238 247 255 255 … 251 254 253 251]
[235 204 167 148 … 189 208 227 247]
…
[212 162 95 53 … 120 170 205 233]
[245 233 218 210 … 226 238 245 249]
[236 238 242 250 … 250 249 247 246]
[233 232 231 234 … 241 238 238 237]]

[[255 255 255 255 … 255 255 255 255]
[255 255 255 255 … 255 255 255 255]
[255 255 255 255 … 255 255 255 255]
[255 255 255 255 … 255 255 255 255]
…
[255 255 255 255 … 255 255 255 255]
[255 255 255 255 … 255 255 255 255]
[255 255 255 255 … 255 255 255 255]
[255 255 255 255 … 255 255 255 255]]

keremnymn · October 24, 2020, 8:02am

LAST UPDATE:

I’ve solved the problem creating my very own dataset, and adding from the dataset I’ve mentioned in original post. It works nearly perfect.
I couldn’t debug the problem properly but I think there may have been a data leak. Because I used “valid_pct” and it assigns validation data randomly, and when I realized how dataset was formed, I came to conclusion that it was high likely a data leak problem.

As always, this was a data problem as well Thanks for the answers though.

JonathanSum · October 24, 2020, 4:14pm

You said you created our own dataset. Do you mean you wrote your own dataset class and load it to the data loader? If so, where is the tutorial if you can tell me? Thank you.

keremnymn · October 25, 2020, 7:54am

I simply created my dataset writing the letters on papers, taking photos of them, augmenting them and dividing them into traing and valid datasets.

JonathanSum · October 25, 2020, 1:45pm

I guess what you said is you solved it by using pretty much the same data loading code but with the image created by you.
I guess the number of images in the training dataset and the number of images in the validating dataset was not balanced (i.e. 90%vs 10%). Or the numbers of images were not enough in both datasets.