How does this not return the exact same result?

I’m baffled here. When loading a pickled model and running the exact same code and data twice how are the exact same results not returned? Am I missing something obvious here?

I tried this just to see if maybe dropout was still running or something like that. Still gave different results.

df = df[df["model"] == "resnet101_EDDDiffClassification_321_2654"]
learn = load_learner("/home/bob/PycharmProjects/relabeling/EDDDiffClassification/resnet101_EDDDiffClassification_321_2654.pkl", cpu=False)
test_dl = learn.dls.test_dl(["/media/bob/NVMe1/data/" + x + ".png" for x in df["image_name"].values])
preds = learn.get_preds(dl=test_dl)

Running this code twice also returns different results:

learn.predict("/media/bob/NVMe1/data/" + df["image_name"].values[0] + ".png")

Any thoughts @muellerzr?

Do you have any callbacks associated with your Learner object?

Here is the entire learner object used:


What version of fastai are you using?

Could you try learn = learn.to_fp32() before performing inference?

Maybe my training transforms are being run on the test_dl somehow? How do I see which transforms are being run at inference time?

Out of curiosity, can you do the following?

dl = learn.dls.test_dl([what you had for predict])
x, = dl.one_batch()

dl2 = learn.dls.test_dl([what you had for predict])
x2, = dl2.one_batch()

And does x and x2 look the exact same?

Re: transforms, only center cropping (when resize is applicable) and normalizing are performed

They do not appear to be the same. Each time I run it there is a change.

1 Like

It looks like augmentations are being applied at test-time. What augmentations do you have?

In dev at least, I can’t reproduce this. Let me try in your pip version though. This passes:

dls = ImageDataLoaders.from_name_func(
    path, get_image_files(path), valid_pct=0.2,
    label_func=lambda x: x[0].isupper(), item_tfms=Resize(448),

learn = cnn_learner(dls, resnet34, metrics=error_rate).to_native_fp16()

dl1 = learn.dls.test_dl([imgs[0]])
dl2 = learn.dls.test_dl([imgs[0]])

x, = dl1.one_batch()
x1, = dl2.one_batch()


How are you building your DataLoader? As @ilovescience asked, can we see your full list of transforms?

LOL I just realized I am Normalizing twice, but not sure if that’s my problem.

tfms = [*aug_transforms(size=500, flip_vert=True, max_rotate=45, max_warp=0)] + [Normalize()]  

    dblock = DataBlock(
      blocks=(ImageBlock, CategoryBlock),      
      getters=[ColReader('image_name', pref="/media/bob/NVMe1/data/", suff=".png"),  
      item_tfms= [mn] + [RandomCrop(350)],
      batch_tfms=tfms + [Normalize()])

dls = dblock.dataloaders(df, bs=24)

What is mn?

class AlbumentationsTransform(Transform):
    def __init__(self, aug): self.aug = aug
    def encodes(self, img: PILImage):
        aug_img = self.aug(image=np.array(img))['image']
        return PILImage.create(aug_img)
mn = MultiplicativeNoise(p=1, elementwise=True, multiplier=(0.85,1.15))

That’s the ticket. That is exactly what is wrong. This is being applied to both the training and the validation set. I would recommend this version from the albumentations tutorial:

class AlbumentationsTransform(DisplayedTransform):
    def __init__(self, train_aug): store_attr()
    def encodes(self, img: PILImage):
        aug_img = self.train_aug(image=np.array(img))['image']
        return PILImage.create(aug_img)

This one will only apply to the training dataset. (Notice the split_idx)

This version is from here:


Dude! You are a genius! Let me run another model and verify, but I’m assuming you are correct here.

With each run is there any data augmentation happening?