📝 Deep Learning Lesson 2 Notes

[PoonamV (Building Jarvislabs.ai)]

@jeremy can you turn this post into wiki.

[PoonamV (Building Jarvislabs.ai)]

Thanks, added in notes.

[cedric (Cedric Chee)]

Hi Poonam, thank you again for starting this. While we wait for Jeremy to turn this post into wiki, could you please add the common header section to your notes and link back to lesson 1 notes?

I have linked to this notes from lesson 1 notes and from the “lesson 2 official resources and updates”. Thanks.

[PoonamV (Building Jarvislabs.ai)]

Thanks Cedric. will surely do that.

[balnazzar (Andrea de Luca)]

Great notes, thank you fr sharing!

[boy1729 (Vikas)]

Pasting the link here so that if thread is converted into wiki, this is not lost:

Deep Learning ver3 Lesson 2 – Vikas Jha – Medium

[jeremy (Jeremy Howard)]

can you turn this post into wiki.

Done.

[bsalita (Robert Salita)]

If you find that the javascript code, which collects urls into a download file, isn’t producing the file or the file is zero length, you may have an ad blocker or popup blocker that’s interfering. Turn off the blocker, refresh the page and try the javascript again. In my case, uBlockOrigin was causing a problem.

Also the javascript doesn’t seem to work on Windows in Microsoft Edge, Firefox, or a private/incognito window. Can anyone verify? It only works in Chrome.

[Lankinen]

I got my personal notes ready and as soon as I have time I will copy some parts from there to here. If someone is interested to read my notes and finds something interesting, with my permission you can just copy past it to these notes. And again if there is too much information to be shared publicly reply this message and I will delete that post. Thanks again to Jeremy this amazing lesson.

Fast.ai Lesson 2 notes — Part 1 v3 – Lankinen – Medium

[cedric (Cedric Chee)]

Detailed lesson 2 notes by Hiromi.

[abhinow]

While picking a learning rate, when do we pick 3e-5 vs 1e-5 ? There was a part of the video in lesson 2 where Jeremy saw the image and put 3e-5 and I wasn’t sure when 1e-5 would be used against 3e-5

[AshwaniTiwari]

After download the url on the local , I have to uplaod that url on the kaggle . The directory created by the python code is “data/beers/…” , then how to upload the file from local to this directory location , because from the “Add Dataset” option on the kaggel , dosent show me this location. Please help me .

[homofortis (Christo MIrchev)]

Here is the javascript to download the links In case you are using DuckDuckGo instead of Google to search for images:
urls = Array.from(document.querySelectorAll('img.tile--img__img.js-lazyload')).map(el=>decodeURIComponent(el.src.split('=')[1])); window.open('data:text/csv;charset=utf-8,' + escape(urls.join('\n')));

[matejthetree (matej)]

I created SGD for square functions. I hope this is the right place to post it
https://www.kaggle.com/matejthetree/sgd-2?scriptVersionId=15140809
I am bad with plot, so I dont know why it pops two images at the end

[lymenlee (Michael Li)]

I manged to download some images of different Chinese Calligraphy artworks. Plan is to build a Chinese calligraphy style classifier. I did all the work on a Kaggle Kernel and uploaded images as private dataset to be used by the kernel. My question may be quite dumb, but what kind of license should I use for the dataset? Or should I upload to Kaggle at all without breaking some license? I want to share the notebook on Kaggle once I finished, but if license won’t clear, I don’t think I can do that. Anyone here could shed some lights on this topic? Much appreciated!

[gbubs]

Does anyone know why my epochs always start from zero instead of one (when I run learn.fit_one_cycle)?
Also why Jeremy used max_lr slice (3e-5, 3e-4) instead of (1e-5, 1e-4) (after running learn.recorder.plot())?

[amit_js]

def update(): y_hat = x@a loss = mse(y, y_hat) if t % 10 == 0: print(loss) loss.backward() with torch.no_grad(): a.sub_(lr * a.grad) a.grad.zero_()

Can Some one please turn this into a set of equations, Especially the

loss.backward()
 with torch.no_grad():
        a.sub_(lr * a.grad)
        a.grad.zero_()

I’m having a hard time understanding it.

[farid (Farid Hassainia)]

a : is the weight tensor (that also stores its gradient in a.grad ) that our model will determine during the training of our model (y = x @ a)
a.grad is an attribute of the a tensor where the gradient of a is stored

a.grad is calculated after each call of the loss.backward() function
Then a is update like this
a = a - lr * a.grad
which can be written like this
a -= lr * a.grad
And in pytorch, it is written like this
a.sub_(lr*a.grad)
it’s called in-place sub() because it directly updates the a tensor in-place

(by the way, if you see a function that ends with _ like sub_(), it means it’s an in-place version of its correspondent function (like add and add_) : It’s a convention)

Once we finish updating the a tensor, we have to reset a.grad to zero ( a.grad.zero_() ) before calling the next loss.backward() function.

As for the with torch.no_grad(): , we use it to ask pytorch to stop updating (tracking) a.grad (it’s already calculated after the loos.backward() call) while we are updating the a tensor

[vatsalnagda (Vatsal)]

Thanks a lot, i spent like 30 mins trying to figure out why i couldnt download. Appreciate it!

[maxho]

Hi all,

I have one quick question regarding the update() function. Been trying to wrap my head around this concept.

Why do we subtract? Because the gradient tells us if I move the whole thing downwards, the loss goes up. If I move the whole thing upwards, the loss goes down. So I want to do the opposite of the thing that makes it go up. We want our loss to be small. That’s why we subtract.

Question:

How do we know that if we move the whole thing downwards, the loss goes up and vice versa?

Appreciate any insights. Thank you.