Part 1: complete collection of video timelines

Part 1 (2017)
1

As a companion to the post “Part #2: complete collection of video timelines”, please find his twin brother for Part #1 below.
Note: this post is an ensemble of the video timelines created by interns & students of Part #1 in the Wiki; I made some editing to keep the flow consistent between lessons.

The full Part #2 video syllabus is available here:

Lesson 1 video timeline

00:00:00 - Fast AI & the course

00:05:29 - Why Deep Learning is exciting

00:10:51 - Deep Learning setup

00:16:02 - Deep Learning trends and applications

00:20:06 - Starting your AWS instance

00:27:07 - Introduction to Jupyter Notebooks

00:33:43 - Introduction to Kaggle

00:41:14 - Introduction to tmux

00:52:57 - Kaggle Dogs vs. Cats data & general data structuring tips

01:01:01 - Introduction to Markdown

01:02:02 - Introduction to some scientific Python libraries

01:09:23 - Pre-trained models & ImageNet

01:15:15 - VGG model

01:17:08 - Implementing VGG

01:22:14 - Python stack being used

01:23:48 - Theano vs. TensorFlow

01:27:02 - Keras and Theano settings

01:30:20 - Batches

01:34:38 - Finetuning ImageNet VGG16 for Dogs vs. Cats

Lesson 2 video timeline

00:0:09 - Teaching Approach

00:05:22 - How to Ask For Help (Tips)

00:07:10 - How to Ask For Help (Example)

00:08:30 - Class Resources: Wiki

00:09:55 - Class Resources: Forum

00:10:25 - Class Resources: Slack

00:11:20 - Class Survey

00:17:14 - Solution to Dogs vs Cats Redux Competition

00:17:30 - Downloading the Data

00:20:00 - Planning (Overview of Tasks)

00:20:25 - Preparing the Data (Validation and Training Set)

00:22:15 - Using Vgg16 (Finetune and Train)

00:22:48 - Submitting to Kaggle

00:30:30 - Competition Evaluation Metric: Log Loss

00:37:18 - Experiment: Running More Epochs

00:40:37 - Visualizing Results

00:47:37 - Introducing the Kaggle State Farm Competition

00:50:29 - Question: Will ImageNet Finetuning Approach work for CT Scans?

00:53:10 - Lesson 0 Video, Convolutions

00:54:09 - Why do we do finetuning?

00:54:43 - What do CNNs learn?

01:03:30 - Deep Neural Network in Excel

01:07:54 - Initialization

01:14:08 Linear Model from Scratch

01:15:10 - Loss function

01:15:49 - Update function

01:24:40 Question: What if you don’t know derivative of functions?

01:25:37 Linear Model in Keras

01:29:58 Linear Model with CNN Features for Dogs Vs Cats Redux

01:44:12 Introducing Activation Functions

01:46:51 Universal Approximation Theorem

01:48:20 Review: Vgg16 Finetuning

Lesson 3 video timeline

00:00:10 - How to use the provided notebooks

00:08:48 - Video of CNN visualization

00:13:11 - CNN review

00:26:34 - VGG review

00:30:13 - Max Pooling review

00:32:12 - CNNs Q&A

00:42:32 - Softmax Function

00:49:40 - SGD review

00:53:10 - More CNNs Q&A

00:59:12 - Finetuning Review

01:12:52 - Underfitting and Overfitting

01:28:42 - Approaches to reducing overfitting

01:31:17 - Data Augmentation

01:39:55 - Batch Normalization

01:48:50 - End-to-End Model Building Process for MNIST

01:57:17 - Ensembling

Lesson 4 video timeline

00:00:0 - CNN review (excel)

00:11:28 - SGD (excel)

00:11:43 - CNN/SGD Q&A

00:26:31 - Visualizing SGD in 2D and 3D

00:28:53 - Visualizing and explaining Momentum in 3D

00:32:20 - Momentum

00:34:35 - Dynamic Learning Rates and Adagrad

00:41:15 - RMSprop

00:46:14 - Adam

00:49:00 - Eve

00:53:52 - Jeremy’s approach to automatic learning rate annealing

00:56:57 - Jeremy’s solution to Kaggle’s “State Farm Distracted Driver Detection”

01:22:05 - Knowledge Distillation (Geoffrey Hinton, Jeff Dean: distilling the knowledge in a Neural Network)
https://youtu.be/V2h3IOBDvrA?t=1h22m5s

01:22:50 - Introduction to Semi-Supervised Learning

01:23:45 - Pseudo-Labeling

01:25:35 - Jeremy’s Kaggle solution Q&A

01:36:01 - Collaborative Filtering

01:51:45 - Collaborative Filtering Q&A

01:58:26 - Collaborative Filtering (continued)

Lesson 5 video timeline

00:00:01 - Tips to get 98.94 acc on Cats and Dogs Redux

00:01:55 - Introducing Batch Normalization into a Pre-Trained Model
& Batch Norm Review + using Batch Norm with VGG

00:10:00 - Collaborative Filtering & Bias Model

00:13:45 - Adding regularization to loss function

00:15:40 - Analyzing Parameters
& Bias + Latent Factors + PCA

00:23:40 - Keras Functional API
& An Aside on Embeddings Functions

00:34:00 - Natural Language Processing
& Sentiment Analysis

00:44:30 - Single hidden layer model

00:56:00 - CNN model & Aside on 1-Dimensional Convolutions

01:12:00 - Unsupervised Learning for Word Embeddings
& Visualizing Word Embeddings

01:31:00 - Using Glove for sentiment analysis

01:36:00 - Multi-Size CNN’s

01:43:06 - Recurrent Neural Network (RNN) & the Need for RNN’s

  • Thinking about Neural Networks as Computational Graphs

01:59:00 - RNN example code for words prediction

Lesson 6 video timeline

00:00:01 - Pseudo-labeling

00:01:15 - MixIterator introduction

00:06:57 - Review: Embeddings

00:08:10 - Embeddings example: MovieLens Data Set

00:13:30 - Word embeddings example: Green Eggs and Ham

00:15:33 - RNNs

00:20:00 - Visual vocabulary for representing neural nets

00:22:56 - 3 kinds of layer operations

00:25:30 - Building first char-RNN in Keras

00:27:28 - Predict 4th character from previous 3

00:38:45 - Generalize first char-RNN formulation: Predict char n from chars 1 to n-1

00:42:20 - RNN from standard Keras dense layers

00:48:25 - Initialization for hidden to hidden dense layer (identity matrix)

00:51:36 - Alternative char-RNN formulation: Predict chars 2 to n using chars 1 to n-1 (sequence to sequence)

01:02:08 - Stateful model with Keras (long-term dependencies)

1:04:30 - Exploding gradients/activations

01:05:55 - LSTM introduction

01:12:07 - Use of TimeDistributed

01:16:50 - Experiments with stacked LSTM

01:23:01 - Build RNN in Theano

01:25:46 - Aside: “loss=sparse_categorical_entropy” alternative to one-hot encoding of output

01:27:30 - Aside: One-hot sequence model with Keras

01:28:50 - Theano overview

01:29:50 - Theano concepts: Variable

01:35:50 - “theano.scan” operation (RNN steps)

01:39:47 - Scan calls step function

01:43:20 - Theano error/loss

01:43:48 - “theano.grad” calculate derivatives

01:44:43 - “theano.function”

01:49:06 - Lesson goals, plans

01:50:15 - In-class questions

01:56:59 - Tip: Exploring layer definitions in keras

02:01:05 - Tip: shift-tab

02:01:40 - Tip: Python debugger in Jupyter notebook

Lesson 7 video timeline

TBD

  • CNN architectures: resnet, inception, fully convolutional net, multi input and multi output nets;
  • Localization with bounding box models and heatmaps;
  • Using larger inputs to CNNs;
  • Building a simple RNN in pure python;
  • Gated recurrent units (GRUs), and how to build a GRU RNN in theano
11 Likes
2

I’m looking for the exact lesson and time where Jeremy presents the “Distilling the Knowledge in a Neural Network” paper written by Hinton and Dean, I missed it :spy:
https://arxiv.org/abs/1503.02531

If you have the info, please post it here so I can update the list.

3

I found the lesson and time for Knowledge Distillation in Lesson 4, and updated the post accordingly.

01:22:05 - Knowledge Distillation (Geoffrey Hinton, Jeff Dean: distilling the knowledge in a Neural Network)

4

This is really great, thanks. Have you considered adding to wiki?