Lesson 12 (2019) discussion and wiki

[rachel (Rachel Thomas)]

Lesson resources

• Lesson video
• Swift for TF lesson signup (share with your friends! Signup not required for people in the full course already.)

Software requirements

• Some nbs of this lesson also currently require pytorch-nightly if yours got de-installed see this.
• Notebook 10c (and subsequent) requires the NVIDIA apex python library. In the environment you created for fastai go to the fastai directory and run pip install git+https://github.com/NVIDIA/apex.

Papers

• L2 Regularization versus Batch and Weight Normalization
• Norm matters: efficient and accurate normalization schemes in deep networks
• Three Mechanisms of Weight Decay Regularization
• Nesterov’s Accelerated Gradient and Momentum as approximations to Regularised Update Descent
• Adam: A Method for Stochastic Optimization
• Reducing BERT Pre-Training Time from 3 Days to 76 Minutes

Blogs

• Visualizing RNNs, by Josh Varty
• Understanding LSTM networks, by Chris Olah

Notes and other resources

• Annotated notebooks for Lessons 8 - 12
• Benchmarking Keras and PyTorch Pre-Trained Models
• Lesson 12 Notes @Lankinen
• Lesson 12 notebooks annotated with video links

AMA about Swift at the end of class

Post your questions in this separate thread:

Ask Jeremy & Chris anything about Swift 4 TensorFlow

Post your questions for Jeremy and Chris Lattner about Swift 4 TensorFlow (which will be covered in lessons 13 and 14). Questions with the most upvotes will be answered at the end of class this evening. Background: fast.ai posts on Swift fast.ai Embracing Swift for Deep Learning High Performance Numeric Programming with Swift: Explorations and Reflections And on why you need to keep learning new languages: You should assume that whatever specific libraries and software you learn today will …

[PierreO (Pierre Ouannes)]

Kind of a tangential question but since Jeremy mentioned it: do you have any tips on debugging Deep Learning models?

[paul (Paul M)]

Has mixup been successfully used in NLP yet?

[sgugger]

Not that I know of, but you should definitely try

[Vishucyrus (Vishal Pandey)]

The Audio module won’t be covered today ???

[rachel (Rachel Thomas)]

Last week we shared an updated schedule. The audio module will be covered in an extra session that will be livestreamed once the course ends. We had more material than will fit in the 7 weeks of the course.

Edited to add: the dates of the extra sessions have not been set yet.

[yeldarb (Brad)]

Has anyone tried mixup and normal augmentation like rotation/zoom?

Seems like they’d be different augmentations that could be used together.

[mcleavey (Christine)]

I tried playing around with mixup on NLP embeddings this past week and from early experimenting it seems to work well (maybe someone else has spent more time on it already!)

[sgugger]

We tried yes, and it’s the same as mixup without normal augmentation in our experiments.

[yeldarb (Brad)]

Do you have an intuition for why that might be?

[edwardeasling]

How broadly can we apply mixup? Could you use it on an image regression problem?

[sgugger]

Mixup is much more powerful in terms of data augmentation, so it more or less erases everything else.

[benjmann]

on mac you can just type ctrl cmd space and type the name of the letter (ie ‘gamma’)

[sgugger]

As long as there is a way to mixup your labels, you can try. It has been wildly experimented in single label classification, not so much in other areas.

[alenas (Alena Harley)]

it seems mixup forces the model to behave linearly between training classes, why is this desirable?

[sgugger]

I’d reverse it into, why is this not desirable?

[alenas (Alena Harley)]

adversarial attacks proofing I guess

[iyersathya (Sathya Iyer)]

What about backprop with new loss?

[sgugger]

Some researchers actually found it helps against adversarial attacks (for a generalized version of Mixup): here.