Bi tempered Logistic Loss

Part 1 (2020)
1

Hi Everyone,

I am training a Multiclass classifier model and want to use Bi tempered logistics loss function. Currently, it is not available in fastai library. Can anybody guide me on how I can integrate it in fastai using PyTorch? I am a beginner so I don’t have much idea. It would be great if you guys can help me out.

2

Hi Radhica,

Assuming you have the pure PyTorch code for “Bi tempered logistics loss”…

  • Place it into a named function in your notebook that takes the output of your model and the target as parameters, and returns a float of the loss.

  • PyTorch will automatically take care of differentiating the loss function and backpropagation through the model.

  • You can manually test that everything works by getting one batch, and trying
    lossfn(model(batch[0]), batch[1])

  • When you create the Learner, pass your loss function for the loss function parameter

HTH, Malcolm
:slightly_smiling_face:

3

An easier way:
If you search for “Bi tempered logistic loss PyTorch", you will find that someone has already done the implementation and testing. Then you only need to bring their function into your notebook and pass it when creating the Learner.
:slightly_smiling_face:

1 Like
4

Hi Malcolm,

Thank you so much for your reply.

I copy-pasted the PyTorch code and called it in learner. However i am having a doubt about what do I pass in activation when I call this in Learner.
learn=cnn_learner(dls,resnet50,
loss_func=bi_tempered_logistic_loss(activation=?,t1=0.2,t2=1,labels=dls.vocab))

Below is the PyTorch code from bi-tempered-loss-pytorch/bi_tempered_loss_pytorch.py at master · fhopfmueller/bi-tempered-loss-pytorch · GitHub.

def bi_tempered_logistic_loss(activations, labels, t1, t2, label_smoothing=0.0, num_iters=5):

"""Bi-Tempered Logistic Loss with custom gradient.
Args:
activations: A multi-dimensional tensor with last dimension `num_classes`.
labels: A tensor with shape and dtype as activations.
t1: Temperature 1 (< 1.0 for boundedness).
t2: Temperature 2 (> 1.0 for tail heaviness, < 1.0 for finite support).
label_smoothing: Label smoothing parameter between [0, 1).
num_iters: Number of iterations to run the method.
Returns:
A loss tensor.
"""

if label_smoothing > 0.0:
    num_classes = labels.shape[-1]
    labels = (1 - num_classes / (num_classes - 1) * label_smoothing) * labels + label_smoothing / (num_classes - 1)

probabilities = tempered_softmax(activations, t2, num_iters)

temp1 = (log_t(labels + 1e-10, t1) - log_t(probabilities, t1)) * labels
temp2 = (1 / (2 - t1)) * (torch.pow(labels, 2 - t1) - torch.pow(probabilities, 2 - t1))
loss_values = temp1 - temp2

return torch.sum(loss_values, dim=-1)
5

Hi Radhica,

The loss function receives activations (the output of the model) and labels. These are provided automatically by the fastai training loop.

The simplest usage is to define your own my_bi_tempered_logistic_loss(activations,labels) whose body passes these two parameters along with fixed parameters t1, t2, etc. to bi_tempered_logistic_loss, and returns its result. The would be called a “partial function”.

Then to cnn_learner, simply pass the name my_bi_tempered_logistic_loss as the loss function.

The situation is confusing because PyTorch loss functions are created for example by a call to nn.MSELoss(…). In your case, my_bi_tempered_logistic_loss is directly the loss function. Just remember, loss_func is itself a function that takes activations and labels, and returns losses.

:slightly_smiling_face:

1 Like
6

Hi Malcolm,
Thank you for explaining in such detail. I was able to implement it and it works like magic!!!

7

I am glad you got it to work!