Model accuracy lower after loading

I am building a text classification model.

When I first trained the model and had it make predictions on the validation set, I got an accuracy of 82.2%. However, upon restarting my jupyter notebook kernel and loading it, I get 69.6% accuracy instead.

I run the following to save the model:
learn.save('modelname')

(learn is a text classifier learner)

When I load the model, I run the following cell:

# Language model data
data_lm = TextLMDataBunch.from_folder(DATA_PATH)
# Classifier model data
data_clas = TextClasDataBunch.from_folder(DATA_PATH, vocab=data_lm.train_ds.vocab, bs=45)

learn = text_classifier_learner(data_clas, AWD_LSTM)

learn.load('modelname')
learn.load_encoder('encodername')
learn.model.eval()

Does anyone know what the problem is here? Is there something I’m doing wrong or is this a bug?

1 Like

I am facing the same issue. I saved the model after training it gave 79% acc and now when I load it in a different notebook, it gives 23% acc.

dls_clas = TextDataLoaders.from_df(df,
text_col = ‘text’,
label_col = ‘target’,
valid_col=‘is_valid’,
bs = 48,
text_vocab = dls_lm.vocab)

learn = text_classifier_learner(dls_clas, AWD_LSTM, drop_mult = 0.5, metrics = accuracy).to_fp16()

load our saved encoder

learn = learn.load_encoder(‘best_lm’)
learn = learn.load(‘predict_desc’)

Were you able to solve this issue?

Same here, the probabilities of my classification task are dramatically different after loading the model:

f1score = F1Score(average="micro")
cbs = [SaveModelCallback(fname="pcb-resnet18")]
learn = cnn_learner(dls, resnet18, metrics=[accuracy, f1score],
               loss_func=LabelSmoothingCrossEntropyFlat(weight=class_weight, eps=0.2), 
               cbs=cbs,
               path="checkpoints", 
               )
learn.fine_tune(5)
predictions, labels, loss = learn.get_preds(with_loss=True)
predictions[0]

tensor([0.4348, 0.0642, 0.0854, 0.0242, 0.0280, 0.0658, 0.0260, 0.0390, 0.0055,
0.0615, 0.0404, 0.0352, 0.0365, 0.0187, 0.0348])

learn = cnn_learner(dls, resnet18, metrics=[accuracy, f1score],
               loss_func=LabelSmoothingCrossEntropyFlat(weight=class_weight, eps=0.2), 
                path="checkpoints",
               )
learn.load("pcb-resnet18")
predictions_new, *_ = learn.get_preds(with_loss=True)
predictions_new[0]

tensor([4.8140e-30, 0.0000e+00, 1.0000e+00, 2.1570e-17, 0.0000e+00, 0.0000e+00,
0.0000e+00, 0.0000e+00, 0.0000e+00, 2.6063e-16, 0.0000e+00, 0.0000e+00,
0.0000e+00, 0.0000e+00, 1.1448e-28])

This is fastai-2.3.1. Any idea what I’m doing wrong would be greatly appreciated.

Answering myself, the resolution is to call learn.model.eval() after learn.load() to put the model into evaluation mode (fix batchnorm and dropout layers).

This is not the first time I stumble over this and quite obviously I’m not the only one to be tripped by forgetting model.eval(). My question then is, why does a library as good as fastai at abstracting away standard and boilerplate code not automatically set the model into evaluation mode when .get_preds()is called? Doesn’t getting predictions imply that I want to evaluate my model instead of training it? Is this behavior on purpose?

3 Likes