Model gives generic answer on new images (works on testing and training data)

I’ve created a model based on Lessons 1 and 2 that I trained with images from a kaggle-Dataset on chest X-Rays, divided into the categories Normal and Pneumonia. With the techniques taught in the lessons (image augmentation, RandomResizedCrop) my model got 100% accuracy (20 Normal and 17 Pneumonia correctly classified). However, if I download a random chest X-Ray from the Internet and let the model classify it for me, it always classifies it as pneumonia images. So it’s not like a random guess, it gets all the pneumonia images right and all the normal X-Rays wrong. Does anyone have an idea what the reason could be?

(Interestingly but maybe not helpfully, it seems to classify pictures that ar not X-Rays at all into the normal category).

My code is:
!pip install -Uqq fastbook
import fastbook
from fastbook import *

path = ‘/content/gdrive/MyDrive/TechLabs/X-Ray’

dls = ImageDataLoaders.from_folder(path, valid_pct=0.2, item_tfms=RandomResizedCrop(224, min_scale=0.7), batch_tfms=aug_transforms(mult=1, do_flip=False, max_rotate=5, max_warp=0.1))

learn = cnn_learner(dls, resnet34, metrics=error_rate)


img = PILImage.create([0])

Any advice appreciated