Lesson 8 - Official topic

[sgugger]

No weights were set to zero. Dropout is applied to activations.

[jwuphysics (John F. Wu)]

A general version of dropout was also proposed much earlier (Hanson 1990) but is rarely cited.

You can find many more instances of these “we or someone else did it first” claims on Schmidhuber’s blog, and here’s a rebuttal from Hinton.

@jcatanza

Hanson’s paper seems to be about adding stochastic noise to improve convergence.

EDIT: rather than adding another post I’m responding here I’m inclined to agree that it’s a bit of a stretch (in addition to some of the other claims on that blog post). However, Hanson did also co-author another paper in 2018: Dropout is a special case of the stochastic delta rule: faster and more accurate deep learning.

[ilovescience (Tanishq)]

Thanks for sharing. It seems more and more that Schmidhuber and his team or other groups did everything already but that’s a discussion for another day!

[nareshr8 (Naresh)]

Will back propagation not update weights twice as it is referred twice in the same network?

[quantum (David Wolpe)]

Why does NLP not use CNNs? Is it because we can’t ‘label’ a body of text?

[ilovescience (Tanishq)]

Here is the questionnaire for chapter 12. It is still a work in progress. Feel free to contribute!:

Fastbook Chapter 12 questionnaire (wiki)

Here are the questions: If the dataset for your project is so big and complicated that working with it takes a significant amount of time, what should you do? Perhaps create the simplest possible dataset that allow for quick and easy prototyping. For example, Jeremy created a “human numbers” dataset. Why do we concatenate the documents in our dataset before creating a language model? To use a standard fully connected network to predict the fourth word given the previous three words, wh…

[init_27 (Sanyam Bhutani)]

End of part 1

[gamino (Ariel Gamiño)]

Thank you for pulling off an amazing class under historical circumstances! Thank you, Rachel, Sylvain, and Jeremy! Thank you, Thank you, Thank you!! And thank you for making us wear masks!

[jcatanza (Joseph Catanzarite)]

Hanson’s paper seems to be about adding stochastic noise to improve convergence.

[radikubwa (M Ben)]

You have change the dimensionality in CNN to a 1D CNN. https://pytorch.org/docs/stable/nn.html. 1D CNNs capture the order of the text as well. I have seen them as way of prepossessing.

[pinaki]

Are there pointers for easier research papers to implement for DL starters ?

[init_27 (Sanyam Bhutani)]

I’ve also created a Wiki: 2020 ML Interviews Resources & Advice(s), please contribute!

[marii (Molly Beavers)]

Definitely suggest kaggle competitions. They can have data separated in other ways besides random split.

Good to get in project groups as well.

[Albertotono]

Will be released a certification, or something to rember this great journey, signed by Jeremy?

[jcatanza (Joseph Catanzarite)]

Applying CNNs to NLP is apparently a thing, at least before transformers!

[ilovescience (Tanishq)]

End of the class

Thanks for a great course amid this pandemic!

I learned a lot even though I went through last year’s course.

I hope Part 2 is coming up soon!

[radikubwa (M Ben)]

try https://paperswithcode.com/

[rachel (Rachel Thomas)]

Mikaela from the USF Data Institute will distribute certificates to those registered for the course (although unfortunately they are not signed).

[erinjerri (Erin Pangilinan)]

Thank you @jeremy and @rachel as always, awesome new material in the fastbook! Can’t wait to get the official O’Reilly version. I’m still working on COVID19 and will talk to you all again soon!

[marii (Molly Beavers)]

Especially the big “Browse State of the Art” button.