Calibrating your network to get better probability estimates

This is a topic for people interested in calibrating their networks to get better probability estimates in classification.

TL;DR: “Neural networks tend to output overconfident probabilities. Temperature scaling is a post-processing method that fixes it. Can we do temperature scaling within the fastAI framework?”

Motivation

Deep network are often overconfident in their predictions. Temperature scaling (TS) works really well to solve this problem and it can be performed as a simple post-processing step. This paper describes the problem of miscalibration, as well as the TS method in detail:

There’s also this video where one of the authors describes the contents of the paper really clearly:
https://vimeo.com/238242536

Implementation of TS in fastAI

I’m not sure how to perform TS on a pertained model within the fastAI framework. Theres is PyTorch code set the temperature in a pre-trained model here, which is a good start:

Here’s the code to run calibration on a pre-trained model:

from temperature_scaling import ModelWithTemperature

orig_model = ... # create an uncalibrated model somehow
valid_loader = ... # Create a DataLoader from the SAME VALIDATION SET used to train orig_model

scaled_model = ModelWithTemperature(orig_model)
scaled_model.set_temperature(valid_loader)

However, I’m not sure how to obtain the required valid_loader in the fastAI framework (without going ‘lower-level’ into full-on PyTorch).

For now I was using a standard high-level training setting in fastAI:

# Load data
data = ImageDataBunch.from_folder(path)

# Train
learn = cnn_learner(data, models.resnet34, metrics=accuracy)
learn.fit_one_cycle(10)
learn.save('model1')

As far as I can tell, the saved model1 is what we need as orig_model for TS. Anyone can think of a way to obtain valid_loader?

Thank you for taking the time to read this!

1 Like

You can use data.train_dl.dl or data.valid_dl.dl which are of the type PyTorch DataLoader

Docs: vision.data

1 Like

Hi Mostafa,

Thanks for the help. I did as you suggested, however I get an error. Here’s what I did:

from temperature_scaling import ModelWithTemperature

# Load data
data = ImageDataBunch.from_folder(path)

# Train
learn = cnn_learner(data, models.resnet34, metrics=accuracy)
learn.fit_one_cycle(10)
learn.save('model1')

orig_model = ".../model1.pth" # create an uncalibrated model somehow
valid_loader = data.valid_dl.dl # Create a DataLoader from the SAME VALIDATION SET used to train orig_model

scaled_model = ModelWithTemperature(orig_model)
scaled_model.set_temperature(valid_loader)

And here’s the error I get:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-d4da6fd75710> in <module>()
      6 
      7 scaled_model = ModelWithTemperature(orig_model)
----> 8 scaled_model.set_temperature(valid_loader)

/content/temperature_scaling/temperature_scaling.py in set_temperature(self, valid_loader)
     46             for input, label in valid_loader:
     47                 input = input.cuda()
---> 48                 logits = self.model(input)
     49                 logits_list.append(logits)
     50                 labels_list.append(label)

TypeError: 'str' object is not callable

I suspect that is because valid_pct default value in from_folder is None. Try setting it to .2 – data = ImageDataBunch.from_folder(path, valid_pct=.2) or use the train loader instead.

After you get the temperature scale by calling scaled_model.set_temperature(valid_loader) implement your own modified softmax function softmax = e^(z/T) / sum_i e^(z_i/T) and pass the model logits to it. This should give you the calibrated predications.

# softmax = e^(z/T) / sum_i e^(z_i/T)
def softmax_calibrated(x, scale=1.):
    if x.ndim == 1:
        x = x.reshape((1, -1))
        
    max_x = x.max(dim=1)[0].reshape((-1, 1))
    exp_x = torch.exp((x - max_x) / scale)

    return exp_x / exp_x.sum(dim=1).reshape((-1, 1))


data_loader = data.test_dl.dl
data_length = len(data_loader)
for batch, (images, labels) in enumerate(data_loader, 1):
    print(f'Step [{batch}/{data_length}], processed {len(labels)} images')

    images = images.to(device)
    labels = labels.to(device)

    logits = learner.model(images)
    preds = softmax_calibrated(logits)
    preds_calibrated = softmax_calibrated(logits, scale=10.699)

    print(f'preds {preds}')
    print(f'preds_calibrated {preds_calibrated}')
1 Like

To solve the

TypeError: 'str' object is not callable

error you simply have to replace

orig_model = ".../model1.pth

with

orig_model = load_learner('.../', 'model1').model

and everything else works fine!

Help me!! I followed the instructions above and got the error:

Our model is text_classifier.
Thanks …

@wildner and @mgazar I am trying to calibrate a fastai tabular model but get into some trouble:

STEP1: I train a fastai tabular classifier

# define path
path = untar_data(URLs.ADULT_SAMPLE)

# load data
df = pd.read_csv(path/'adult.csv')

# simple split data into train & valid
valid_idx = range(len(df)-2000, len(df))

# define local variables
dep_var = 'salary'
cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']
n_epochs = 2

# data preprocessing
procs = [FillMissing, Categorify, Normalize]

# prep data for tabular_learner()
data = TabularDataBunch.from_df(path, df, dep_var, valid_idx, procs=procs, cat_names=cat_names)

#metrics
f1=FBeta()
precision = Precision()
recall = Recall()
metrics=[accuracy, precision, recall, f1]

# train a classifier
clf = tabular_learner(data, layers=[200,100], emb_szs={'native-country': 10}, metrics=metrics)
lr = 1e-2
clf.fit_one_cycle(n_epochs, moms=(lr*0.01,lr))

STEP2: I use ModelWithTemperature and .set_temperature() from here: https://github.com/gpleiss/temperature_scaling/blob/master/temperature_scaling.py which you can copy and paste the code into your colab notebook cell.
To address an error, I modified one line of code in def set_temerature(self, valid_loader) here: I replaced input.cuda() by input = [x.cuda() for x in input]

STEP3 apply temperature method:

scaled_clf = ModelWithTemperature(clf.model)
scaled_clf.set_temperature(data.valid_dl)

The error message is:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-19-d54248adb865> in <module>()
----> 1 scaled_clf.set_temperature(data.valid_dl)

1 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

TypeError: forward() missing 1 required positional argument: 'x_cont'

Any advice would be much appreciated!

How’d you go with this one in the end?