Accuracy drop after loading model

fastai 2.7.16 py_0 fastai
I have looked at other posts (Post 1 & Post 2) with people having the same problem as I have, but their solution doesn’t seem to work in my case :c

I am using a pre-trained resnet50 for classifying images into 3 categories.
My program checks the models directory to see if there is an existing model. If there isn’t it will train a new one, if there is, it will use that.
When training a new model and immediately using that after, it runs just fine, and sorts the images into their respective categories. But when loading a model, which has already been trained (and previously worked just fine), it only places the images into the first category. Only on very rare occasions will it pop out a couple images into another category.

def create_new_model(trainset_path, train=True):
    
    # Create DataLoaders from the folder structure
    #print(trainset_path)
    dls = make_dls(trainset_path) # This function returns a dataloader

    # Create and train the model
    learn = vision_learner(dls, resnet50, metrics=accuracy, pretrained=train)

    if train: learn.fine_tune(3) # I don't want to train the model, if I can load one much faster.

    return learn


# This function outputs a learner which is used in another script like this: "learn.predict(img)"
def get_model(trainset_path : pathlib.PosixPath):
    # Path to the file
    model_path_pth = trainset_path / "models/model.pth"
    
    # If the path exists
    if model_path_pth.exists():
        print(f"Loading model from {model_path_pth}.")
        learn = create_new_model(trainset_path, train=False)
        #learn = load_learner(model_path_pth)
        learn.load("model")
        learn.model.eval() # <--- Here I am trying the solution from the two other threads having the same issue
    else:
        print(f"Creating new model at {model_path_pth}.")
        learn = create_new_model(trainset_path)
        learn.save("model")

    return learn

I have also done learn.summary() and compared the trained version and the loaded version using meld, and the only difference in them is the location of the optimizer: