Lesson 10 Discussion & Wiki (2019)

[tanyaroosta]

It was mentioned in the lecture that doing batch norm with RNN/LSTM is not straight forward, but there are a few tensorflow implementations and a couple of papers discussing how to do it. Does it mean it’s possible but not advisable to use batch norm with RNN, or that these papers are not getting it right?

[jeremy (Jeremy Howard)]

It was mentioned in the lecture that doing batch norm with RNN/LSTM is not straight forward, but there are a few tensorflow implementations and a couple of papers discussing how to do it

Can you link to the papers?

[tanyaroosta]

MargaritSubramaniam.pdf

https://openreview.net/pdf?id=r1VdcHcxx

Recurrent Batch Normalization

[jeremy (Jeremy Howard)]

MargaritSubramaniam.pdf

That’s a student project where it looks like they couldn’t get anything to work.

Recurrent Batch Normalization

We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks

They say “we recommend using separate statistics for each timestep to preserve information of the initial transient phase in the activations”. That’s the thing we were trying to avoid. It would be OK for short sequences that don’t vary in length too much, but it seems difficult to do that in practice if you need to handle (say) sequences of 3000 words or so and which vary in length a lot (like in IMDb).

[fabris (Fabrizio)]

Just a proposal!! Since I see almost no discussions about study groups despite Jeremy’s post, why don’t we use some kaggle competition like the one linked above so as to motivate people to cooperate ?? Sharing a common goal could be very useful. Moreover, let us make randomly study groups! But, of course, this latter it is up to the administrators if this idea is widely supported.

[ste (Stefano Giomo)]

That is the post on colorful histograms

The "colorful" dimension

TL;DR The idea of the colorful dimension is to express with colors the mean and standard deviation of activations for each batch during training. Vertical axis represents a group (bin) of activation values. Each column in the horizontal axis is a batch. The colours represent how many activations for that batch have a value in that bin. [image] NB: Actual plot shown by Jeremy in the lesson are much more curated and focused. This post is about the original idea and my attempt to implement it. D…

[Pomo (Malcolm McLean)]

To clarify my understanding…

• After sigmoid, you’d apply cross entropy to get the loss?

• Sigmoid + cross entropy loss is what multi-label classification uses by default?

[sgugger]

No, sigmoid + binary cross entropy is binary cross entropy, and it’s not the default for multi-label classification. That default is softmax + cross entropy.

[jeremy (Jeremy Howard)]

The edited video has now been added to the top post.

[Pomo (Malcolm McLean)]

Arrgh - not clarified yet.

• Could you show the sequence of functions that turns activations into a loss number, using “binomial loss”?

• When I run the lesson3-planets multi-label example, the model ends with Linear and the loss function is FlattenedLoss of BCEWithLogitsLoss(). This is defined as sigmoid followed by binary cross entropy loss.

Thanks for sorting this out.

[charming (Charm)]

I have some trouble understanding the use of register_buffer().
My questions are:

1. When should I register a buffer? For what sort of Variables and for which not?
2. Could someone provide me with a simple example and code snippet of using register_buffer()?

[LessW2020 (Less)]

From reading about this in the Pytorch forums, here’s some info for your first question:
" If you have parameters in your model, which should be saved and restored in the state_dict , but not trained by the optimizer, you should register them as buffers.
Buffers won’t be returned in model.parameters() , so that the optimizer won’t have a chance to update them."
I hope to do some work with them tomorrow and if so will post a code snippet (assuming somone else doesn’t beat me to it

[LessW2020 (Less)]

Regarding question 2 - here’s the code for batchnorm and you can see how they register params vs buffers. Params are learnable (i.e. gradient) vs buffers are not, so that’s the main difference:

def __init__(self, num_features, eps=1e-5, momentum=0.1, affine=True,
                 track_running_stats=True):
        super(_BatchNorm, self).__init__()
        self.num_features = num_features
        self.eps = eps
        self.momentum = momentum
        self.affine = affine
        self.track_running_stats = track_running_stats
        if self.affine:
            self.weight = Parameter(torch.Tensor(num_features))
            self.bias = Parameter(torch.Tensor(num_features))
        else:
            self.register_parameter('weight', None)
            self.register_parameter('bias', None)
        if self.track_running_stats:
            self.register_buffer('running_mean', torch.zeros(num_features))
            self.register_buffer('running_var', torch.ones(num_features))
            self.register_buffer('num_batches_tracked', torch.tensor(0, dtype=torch.long))
        else:
            self.register_parameter('running_mean', None)
            self.register_parameter('running_var', None)
            self.register_parameter('num_batches_tracked', None)
        self.reset_parameters()

Hope that helps!

[marcmuc (Marc P. Rostock)]

No, sigmoid + binary cross entropy is binary cross entropy, and it’s not the default for multi-label classification. That default is softmax + cross entropy.

Hmm, do you mean multi-label or multi-CLASS? I think so far
multi-class default is Categorical Cross Entropy using softmax and multi-label default is Binary Cross Entropy using sigmoid. (These are also the defaults in the fastai library based on the labelclass)

So from my understanding of Jeremy in the lecture it would often make sense for real world mutliclass problems to not use softmax but rather the binary cross entropy (multi-label) version and then use thresholds and/or argmax with the results to figure out the single class. In that way we also get the probabilities for the class, undistorted by softmax, in order to be able to differentiate given classes vs. “background”/“no label”, in case probabilities are small for all of the classes. Is this what he meant?

This would finally answer my question asked during v3 part 1 :

Lesson 2: further discussion

Is it better to train only on the correct categories and then have a mechanism that based on very low probabilities across categories will say “none of the above”? (Isn’t this difficult when using softmax, because that will still give you some “winner” category most of the time)

from here.

[radikubwa (M Ben)]

Thanks @deena-b I’ll look into VScode soon. I’ve used vim while writing bash scripts but I kept forgetting the commands all the time and deleting my code. I’ll text you tomorrow.

Yes, those are great takeaways I’ll write that down.

[dreambeats (JamesT)]

I was looking at the new version of the Runner class, and I realised that we may have lost the ability for a callback to return True, is that correct?

Since res is set to False at the start, and we are using the ‘and’ operator, this effectively means that no matter what the callbacks return, res will be ultimately False, right?

[Lankinen]

I remember Graham Neubig saying that batch size is a hyperparameter. Can someone explain that? What is the difference of having batch size of 32 instead of 128 addition to the speed?

[karthik.subraveti]

Might be some basic mistake here. I’m confused in different behaviors in numpy and torch

np.array([10, 20]).var()
25.0

np.array([10, 20]).std()
5.0

torch.tensor([10., 20.]).var()
tensor(50.)

torch.tensor([10., 20.]).std()
tensor(7.0711)

in torch’s case they don’t seem to be taking a mean of the sum of the square of the deviations for the variance. Is this a bug ?

Update:
I digged further into this and looks like there is an arg called “unbiased” and if i set that, it matches numpy.

torch.tensor([10., 20.]).var(unbiased=False)
tensor(25.)

torch.tensor([10., 20.]).std(unbiased=False)
tensor(5.)
If unbiased is False , then the standard-deviation will be calculated via the biased estimator. Otherwise, Bessel’s correction will be used.

[jeremy (Jeremy Howard)]

What confused me a bit was the “binomial” heading in the rightmost excel table, because I’m more familiar with calling it “multi-label binary classification”

Oh silly me - I meant to say “binary” but wrote “binomial” then just read what was there rather than actually thinking about it! Thanks for pointing this out.

[tanyaroosta]

One thing I didn’t quite understand is Jeremy said softmax should not be used, but everyone uses it. What should be used instead? Or did I misunderstand?