Lesson 4 - Official Topic

[erlapi (Emilio)]

Could you please clarify the difference between dataloader and dataloaderS ?

[aho (Alvin)]

Is there a fastai function to figure this out based on the dataset to maximize batch size based on GPU mem?

[sgugger]

As the s imply, dataloaders is an object with several dataloader: one for training and one for validation.

[ilovescience (Tanishq)]

Simply put, DataLoaders is a wrapper for multiple DataLoader's (ex: train and validation dataloader)

[amb]

Why do you stack the 3s and 7s dataset again? On top of each other.

[sgugger]

That is not entirely true, as we have rewritten the Pytorch DataLoader in fastai,

[sgugger]

You want to train your model on 3s and 7s together, or it won’t learn to differentiate between the two of them.

[sgugger]

Not yet no. There was in fastai v1, and you can probably import it while waiting for it to be ported to v2.

[matdmiller (Mat M)]

I don’t believe so. It doesn’t take too much trial and error to figure it out. GPU usage is depending on both your dataset items and the model you choose.

[ilovescience (Tanishq)]

I think this was the one used in fastai v1:

Speed up batch size discovery hundreds of times (to fit into your GPU RAM)

OK, you know how time consuming the bs discovery is, you got to run lr_find/fit and then if bs is too big do it again, then the next stage, and repeat it all together again and again, then the next stage, and meanwhile waiting, and waiting and waiting… Then recently I discovered that all the required GPU memory is setup during the first batch of the first epoch see tutorial. So I thought why do I waste hours waiting for things to finish. I also find the waiting makes me really tired physically.…

[ren]

Broadcasting was used when adding the bias vector to the weight matrix.

Can you think of other parts of the training process where broadcasting is used?

[jona (Jona R)]

In the simplest model of SGD (the function called “train_epoch”, the for loop is based on an iterator “dl”, but that is not passed into the function. How does the function get that variable?

[sgugger]

It’s defined in the notebook.

[yfrancois (Yann François)]

Maths are a bit rusty… Why the name linear? The bias isn’t making the matrix multiplication non-linear?

[marii (Molly Beavers)]

Think of it as y = mx + b

[sgugger]

Technically, the bias makes it affine, but people still often say linear.

[erikg (Erik Gaasedelen)]

y = mx + b. Is still just a linear function. m for the slope. b for just shifting the line up and down.

y = mx^2 + b would be nonlinear because of the ^2

Edit: I never realized that linear was incorrect and should be affine as sylvain notes. I feel betrayed by conventional vocabulary.

[maya (ravisha)]

What’s an affine?

Just a linear with intercept?

[sgugger]

Yes, but don’t get too distracted by the names They are not super important.

[harish3110 (Harish Vadlamani)]

By using the non-linearity, won’t using a function that makes all negative outputs to zero make many of the gradients in the network zero and stop the learning process due to many zero gradients?