This is a topic for people interested in calibrating their networks to get better probability estimates in classification.
TL;DR: “Neural networks tend to output overconfident probabilities. Temperature scaling is a post-processing method that fixes it. Can we do temperature scaling within the fastAI framework?”
Deep network are often overconfident in their predictions. Temperature scaling (TS) works really well to solve this problem and it can be performed as a simple post-processing step. This paper describes the problem of miscalibration, as well as the TS method in detail:
There’s also this video where one of the authors describes the contents of the paper really clearly:
Implementation of TS in fastAI
I’m not sure how to perform TS on a pertained model within the fastAI framework. Theres is PyTorch code set the temperature in a pre-trained model here, which is a good start:
Here’s the code to run calibration on a pre-trained model:
from temperature_scaling import ModelWithTemperature orig_model = ... # create an uncalibrated model somehow valid_loader = ... # Create a DataLoader from the SAME VALIDATION SET used to train orig_model scaled_model = ModelWithTemperature(orig_model) scaled_model.set_temperature(valid_loader)
However, I’m not sure how to obtain the required
valid_loader in the fastAI framework (without going ‘lower-level’ into full-on PyTorch).
For now I was using a standard high-level training setting in fastAI:
# Load data data = ImageDataBunch.from_folder(path) # Train learn = cnn_learner(data, models.resnet34, metrics=accuracy) learn.fit_one_cycle(10) learn.save('model1')
As far as I can tell, the saved
model1 is what we need as
orig_model for TS. Anyone can think of a way to obtain
Thank you for taking the time to read this!