I’m baffled here. When loading a pickled model and running the exact same code and data twice how are the exact same results not returned? Am I missing something obvious here?
I tried this just to see if maybe dropout was still running or something like that. Still gave different results.
df = df[df["model"] == "resnet101_EDDDiffClassification_321_2654"] learn = load_learner("/home/bob/PycharmProjects/relabeling/EDDDiffClassification/resnet101_EDDDiffClassification_321_2654.pkl", cpu=False) learn.model.eval() test_dl = learn.dls.test_dl(["/media/bob/NVMe1/data/" + x + ".png" for x in df["image_name"].values]) preds = learn.get_preds(dl=test_dl) preds
Running this code twice also returns different results:
learn.predict("/media/bob/NVMe1/data/" + df["image_name"].values + ".png")
Any thoughts @muellerzr?
Do you have any callbacks associated with your Learner object?
Here is the entire learner object used:
learn=cnn_learner(dls, torchvision.models.resnet152, metrics=[accuracy], cbs=[ShowGraphCallback]).to_fp16()
What version of fastai are you using?
Could you try
learn = learn.to_fp32() before performing inference?
Maybe my training transforms are being run on the test_dl somehow? How do I see which transforms are being run at inference time?
Out of curiosity, can you do the following?
dl = learn.dls.test_dl([what you had for predict]) x, = dl.one_batch() dl2 = learn.dls.test_dl([what you had for predict]) x2, = dl2.one_batch()
And does x and x2 look the exact same?
Re: transforms, only center cropping (when resize is applicable) and normalizing are performed
They do not appear to be the same. Each time I run it there is a change.
It looks like augmentations are being applied at test-time. What augmentations do you have?
In dev at least, I can’t reproduce this. Let me try in your pip version though. This passes:
dls = ImageDataLoaders.from_name_func( path, get_image_files(path), valid_pct=0.2, label_func=lambda x: x.isupper(), item_tfms=Resize(448), batch_tfms=[*aug_transforms(size=224)]) learn = cnn_learner(dls, resnet34, metrics=error_rate).to_native_fp16() dl1 = learn.dls.test_dl([imgs]) dl2 = learn.dls.test_dl([imgs]) x, = dl1.one_batch() x1, = dl2.one_batch() test_eq(x,x1)
How are you building your
DataLoader? As @ilovescience asked, can we see your full list of transforms?
LOL I just realized I am Normalizing twice, but not sure if that’s my problem.
tfms = [*aug_transforms(size=500, flip_vert=True, max_rotate=45, max_warp=0)] + [Normalize()] dblock = DataBlock( blocks=(ImageBlock, CategoryBlock), getters=[ColReader('image_name', pref="/media/bob/NVMe1/data/", suff=".png"), ColReader('EDD_diff_class')], splitter=ColSplitter(), item_tfms= [mn] + [RandomCrop(350)], batch_tfms=tfms + [Normalize()])
dls = dblock.dataloaders(df, bs=24)
class AlbumentationsTransform(Transform): def __init__(self, aug): self.aug = aug def encodes(self, img: PILImage): aug_img = self.aug(image=np.array(img))['image'] return PILImage.create(aug_img) mn = MultiplicativeNoise(p=1, elementwise=True, multiplier=(0.85,1.15))
That’s the ticket. That is exactly what is wrong. This is being applied to both the training and the validation set. I would recommend this version from the albumentations tutorial:
class AlbumentationsTransform(DisplayedTransform): split_idx,order=0,2 def __init__(self, train_aug): store_attr() def encodes(self, img: PILImage): aug_img = self.train_aug(image=np.array(img))['image'] return PILImage.create(aug_img)
This one will only apply to the training dataset. (Notice the
This version is from here: https://docs.fast.ai/tutorial.albumentations.html#Using-different-transform-pipelines-and-the-DataBlock-API
Dude! You are a genius! Let me run another model and verify, but I’m assuming you are correct here.
With each run is there any data augmentation happening?