Implementing temperature scaling

[wildner]

Hello everyone,
I tired to implement temperature scaling as a post-processing step on my fastai classifier, but ran into some problems.
In case you are interrested in temperature scaling I can suggest another post from this forum that has a great introduction about it:

Calibrating your network to get better probability estimates

This is a topic for people interested in calibrating their networks to get better probability estimates in classification. TL;DR: “Neural networks tend to output overconfident probabilities. Temperature scaling is a post-processing method that fixes it. Can we do temperature scaling within the fastAI framework?” Motivation Deep network are often overconfident in their predictions. Temperature scaling (TS) works really well to solve this problem and it can be performed as a simple post-processi…

Unfortunately the above post was never solved and I had the same error messages as the author. My code:

from temperature_scaling import ModelWithTemperature

# Load data
data = ImageDataBunch.from_folder(path,valid_pct=.2)


orig_model = load_learner('../model1.pth') 
valid_loader = data.valid_dl.dl

scaled_model = ModelWithTemperature(orig_model)
scaled_model.set_temperature(valid_loader)

And I get the error

Traceback (most recent call last):
  File "temperatureScaling.py", line 47, in <module>
    scaled_model.set_temperature(valid_loader)
  File "../temperature_scaling.py", line 48, in set_temperature
    logits = self.model(input)
TypeError: 'Learner' object is not callable

As suggested in the linked forum article I tried setting valid_pct=.2 but that did not solve the problem. It has to do with orig\_model not being callable. Is there a workaround for this?

Thank you for reading!
Good day

[wildner]

Solved it myself. I posted my solution in the linked post above.