NN tutorial for Pytorch

Intro to Machine Learning (2018)
1

Hi,
I’m trying to go over the great tutorial that the FastAI team just added to the Pytorch tutorial sitehttps://pytorch.org/tutorials/beginner/nn_tutorial.html.

I’m having hard time to understand the negative log-likeihood function, I understand from the text it is supposed to be standard Python broadcasting technique, but i’m not familiar how the range function behave inside a list comprehension.

def nll(input, target):
    return -input[range(target.shape[0]), target].mean()

Any help or a reference where to search for an explanation will be great.

Cheers

2 Likes
2

Hi,

The loss function is L(y)=−log(y), which is summed for all the correct classes, and given that the model outputs are in the form of log probabilities, all we need to do is get the log probabilities for the correct classes, which is done by indexing the targets : input[range(target.shape[0]), target], So for each example in the batch (range(target.shape[0])) we get the outputs corresponding to the correct class, and we take the mean over the batch.

this is how I see it.

1 Like
3

could u please explain the line code with more details?

4

Hi,

I found the best way for me to understand it is to reproduce it on excel.

NN%20tutorial_Pytorch
NN tutorial_Pytorch.JPG1758×345 72.5 KB

The preds 2D tensor (64X10) present the Log of the predictions - 10 for each sample (cols) and batch of 64 (rows).

This line of code takes for each row the log of predicted value and the col index is the target value for this row.
for example if the first row represent the number 5, it will take the 5th element from the pred tensor first row - which represent the log of the model estimation (soft-max) that this row is indeed 5).
then it do the same for all rows and average the results.

Since it is the log of the softmax, if it is correct estimation - the estimation will be close to one and the log will be close to zero, which will lead to low Loss.