Unofficial Deep Learning Lecture 5 Notes
Hi All,
Please find the lecture 5 notes. There was a lot of great material covered today, so hoping you guys can flesh out some of the details. I tried to capture as much as I could, but there was a lot of math to absorb near the end of the lecture around the topics of momentum + Adam optimizer.
Hope the notes are useful!
Cheers,
Tim
@jeremy please convert to wiki, so all can contribute? Thanks ahead of time, great lecture today.
Today we are going to start the second part of this course. Now, what we now going to do is go in reverse. We are going to go back over all of those exact same things again but this time we are going to dig into the detail of every one and we are going to look inside the source code of fastai library to see what it’s doing and try to replicate that. So, in a sense like there’s not going to be a lot more best practices to show us like Jeremy have kind of shown us the best best practices that he knows. Jeremy thinks it’s time for us to now build on top of those, to debug those models, to come back to part 2 where we’re going to kind of try out some new things. It really helps to understand what’s going on behind the scenes.
MovieLens
The goal here today is we’re going to try and create a pretty effective collaborative filtering model almost entirely from scratch. We will use PyTorch as a automatic differentiation tool and as a GPU programming tool and not very much else. We’ll try not to use its neural net features. We’ll try not to use fastai library anymore than necessary.
Let’s look at collaborative filtering. We are going to look at Movielens dataset. The Movielens dataset is basically is a list of movie ratings by users.
We go through the Jupyter notebook here.
If you need to download the dataset:
# !wget http://files.grouplens.org/datasets/movielens/mllatestsmall.zip
# !unzip mllatestsmall.zip
Import the libraries:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
import torch
from fastai.learner import *
from fastai.column_data import *
path='mllatestsmall/'
ratings = pd.read_csv(path+'ratings.csv')
ratings.head()
userId  movieId  rating  timestamp  

0  1  31  2.5  1260759144 
1  1  1029  3.0  1260759179 
2  1  1061  3.0  1260759182 
3  1  1129  2.0  1260759185 
4  1  1172  4.0  1260759205 
movies = pd.read_csv(path+'movies.csv')
movies.head()
movieId  title  genres  

0  1  Toy Story (1995)  AdventureAnimationChildrenComedyFantasy 
1  2  Jumanji (1995)  AdventureChildrenFantasy 
2  3  Grumpier Old Men (1995)  ComedyRomance 
3  4  Waiting to Exhale (1995)  ComedyDramaRomance 
4  5  Father of the Bride Part II (1995)  Comedy 
Create subset for Excel
We create a crosstab of the most popular movies and most movieaddicted users which we’ll copy into Excel for creating a simple example. This isn’t necessary for any of the modeling below however.
g=ratings.groupby('userId')['rating'].count()
topUsers=g.sort_values(ascending=False)[:15]
g=ratings.groupby('movieId')['rating'].count()
topMovies=g.sort_values(ascending=False)[:15]
top_r = ratings.join(topUsers, rsuffix='_r', how='inner', on='userId')
top_r = top_r.join(topMovies, rsuffix='_r', how='inner', on='movieId')
# pd.crosstab(top_r.userId, top_r.movieId, top_r.rating, aggfunc=np.sum)
Given some ratings:
Every rating is represented as a matrix cross product (random init)
Note that Microsoft Office for Windows has addins. You can turn on the “solver” in the addins. This will run an iterative solver.
Now we can calculate the loss
And we can then optimize the ORANGE cells based on the loss, BLUE cells  RED cells.
Collaborative Filtering  High Level
Get a validation indexes

wd
 L2 regularization 
n_factors
 embedding sizes
val_idxs = get_cv_idxs(len(ratings))
wd=2e4
n_factors = 50
ratings.csv
userId
movieId

rating
 dependent variable
This movie and user, which other movies are similar to it, which people are similar to this person?
cf = CollabFilterDataset.from_csv(path, 'ratings.csv', 'userId', 'movieId', 'rating')
 size embedding
 index
 batch size
 which optimizer
learn = cf.get_learner(n_factors, val_idxs, 64, opt_fn=optim.Adam)
Run the fit
learn.fit(1e2, 2, wds=wd, cycle_len=1, cycle_mult=2)
[ 0. 0.84641 0.80873]
[ 1. 0.78206 0.77714]
[ 2. 0.5983 0.76338]
math.sqrt(0.776)
0.8809086218218096
preds = learn.predict()
y=learn.data.val_y[:,0]
sns.jointplot(preds, y, kind='hex', stat_func=None);
PyTorch and Linear Algebra: Dot product example
T
= tensor in Torch
a = T([[1.,2],[3,4]])
b = T([[2.,2],[10,10]])
a,b
(
1 2
3 4
[torch.FloatTensor of size 2x2],
2 2
10 10
[torch.FloatTensor of size 2x2])
a*b
2 4
30 40
[torch.FloatTensor of size 2x2]
(a*b).sum(1)
6
70
[torch.FloatTensor of size 2]
Our First Custom Layer in PyTorch
We are now making a basic Torch
class. We are extending the nn.Module
from PyTorch
forward
is a inherited method from the nn.Module
, which is leveraged by other functions in other pytorch libraries

forward
< matrix multiplication 
backwards
< gradients
class DotProduct(nn.Module):
def forward(self, u, m): return (u*m).sum(1)
Create an instance of the class
model=DotProduct()
model(a,b)
6
70
[torch.FloatTensor of size 2]
Prepare the data:
 look at unique values of user_ids
 make a mapping from user_id to a INT (so there’s no skipping of numbers)
 do the same for movie ids
Reminder:

enumerate
 takes in a collection, gives back the index, value for a collection 
lambda
 python keyword for making a temporary function
u_uniq = ratings.userId.unique()
user2idx = {o:i for i,o in enumerate(u_uniq)}
ratings.userId = ratings.userId.apply(lambda x: user2idx[x])
m_uniq = ratings.movieId.unique()
movie2idx = {o:i for i,o in enumerate(m_uniq)}
ratings.movieId = ratings.movieId.apply(lambda x: movie2idx[x])
n_users=int(ratings.userId.nunique())
n_movies=int(ratings.movieId.nunique())
Now lets make a more complicated Embedding Layer

__init__
 this function is called when the object is created. Called the constructor 
obj = myClass()

obj = newClassImade(list)

nn.Embedding
 a pytorch object 
self.u.weight.data
 initializing the user weights to a reasonable starting place 
self.m.weight.data
 initializing the movie weights to a reasonable starting place 
uniform_
 note that_
means ‘inplace’ instead of returning a variable and doing an assignment
for forward
 users,movies = cats[:,0],cats[:,1]
We are pulling both datasets out of the categorical variable.
class EmbeddingDot(nn.Module):
def __init__(self, n_users, n_movies):
super().__init__()
self.u = nn.Embedding(n_users, n_factors)
self.m = nn.Embedding(n_movies, n_factors)
self.u.weight.data.uniform_(0,0.05)
self.m.weight.data.uniform_(0,0.05)
def forward(self, cats, conts):
users,movies = cats[:,0],cats[:,1]
u,m = self.u(users),self.m(movies)
return (u*m).sum(1)
Prep the features and dependent variable
x = ratings.drop(['rating', 'timestamp'],axis=1)
y = ratings['rating']
Pull from a dataframe, with x, y, note that we bind the categorical variables together
data = ColumnarModelData.from_data_frame(path, val_idxs, x, y, ['userId', 'movieId'], 64)
Setup a optimizer

optim
 gives us an optimizer. 
model.parameters()
this is inherited from thenn.Module
. 
1e1
 is the learning rate 
weight decay
 regularization momentum
wd=1e5
model = EmbeddingDot(n_users, n_movies).cuda()
opt = optim.SGD(model.parameters(), 1e1, weight_decay=wd, momentum=0.9)
Note: we are not using a Learner
, we are using the lowlevel PyTorch fit
.
fit(model, data, 3, opt, F.mse_loss)
[ 0. 1.68932 1.64501]
[ 1. 1.08445 1.30609]
[ 2. 0.91446 1.23001]
### Set the learning rate
set_lrs(opt, 0.01)
fit(model, data, 3, opt, F.mse_loss)
[ 0. 0.69273 1.14723]
[ 1. 0.72746 1.1352 ]
[ 2. 0.67176 1.12943]
Bias
min_rating,max_rating = ratings.rating.min(),ratings.rating.max()
min_rating,max_rating
(0.5, 5.0)
Let’s improve our model
Can we squash the ratings between 1 and 5?
With the sigmoid function we can multiply by our target range
F
 its the pytorch functional. its a large library of functions used in deep learning
def get_emb(ni,nf):
e = nn.Embedding(ni, nf)
e.weight.data.uniform_(0.01,0.01)
return e
class EmbeddingDotBias(nn.Module):
def __init__(self, n_users, n_movies):
super().__init__()
(self.u, self.m, self.ub, self.mb) = [get_emb(*o) for o in [
(n_users, n_factors), (n_movies, n_factors), (n_users,1), (n_movies,1)
]]
def forward(self, cats, conts):
users,movies = cats[:,0],cats[:,1]
um = self.u(users)* self.m(movies)
## add in user bias and movie bias
## squeeze is going to be broadcasting, this will replicate a vector
## and add it to the matrix
res = um.sum(1) + self.ub(users).squeeze() + self.mb(movies).squeeze()
## this limits the range of the ratings
res = F.sigmoid(res) * (max_ratingmin_rating) + min_rating
return res
wd=2e4
model = EmbeddingDotBias(cf.n_users, cf.n_items).cuda()
opt = optim.SGD(model.parameters(), 1e1, weight_decay=wd, momentum=0.9)
fit(model, data, 3, opt, F.mse_loss)
[ 0. 0.85056 0.83742]
[ 1. 0.79628 0.81775]
[ 2. 0.8012 0.80994]
What’s actually written in fastai library?
def get_emb(ni,nf):
e = nn.Embedding(ni, nf)
e.weight.data.uniform_(0.05,0.05)
return e
class EmbeddingDotBias(nn.Module):
def __init__(self, n_factors, n_users, n_items, min_score, max_score):
super().__init__()
self.min_score,self.max_score = min_score,max_score
(self.u, self.i, self.ub, self.ib) = [get_emb(*o) for o in [
(n_users, n_factors), (n_items, n_factors), (n_users,1), (n_items,1)
]]
def forward(self, users, items):
um = self.u(users)* self.i(items)
res = um.sum(1) + self.ub(users).squeeze() + self.ib(items).squeeze()
return F.sigmoid(res) * (self.max_scoreself.min_score) + self.min_score
Let’s make a Neural Network (NN) version of this
Let’s look at the Excel.
All the users are compared to all movies.
Why not take the embedding and make a NN out of it?
Mini net
Let’s prototype our Embedding NN

(self.u, self.m) = [get_emb(*o) for o in..
create embeddings 
nn.Linear(n_factors*2, 10, nh)
 create our linear layers 
nh
number of hidden layers 
torch.cat([self.u(users),self.m(movies)], dim=1)
adds the two users and movies together 
F.relu(self.lin1(x)
 linear
 relu 0, or Max; truncates negative values
 we have 1 layer, it is a NN, maybe not ‘deep’ per say
class EmbeddingNet(nn.Module):
def __init__(self, n_users, n_movies, nh=10):
super().__init__()
(self.u, self.m) = [get_emb(*o) for o in [
(n_users, n_factors), (n_movies, n_factors)]]
self.lin1 = nn.Linear(n_factors*2, 10, nh)
self.lin2 = nn.Linear(10, 1)
def forward(self, cats, conts):
users,movies = cats[:,0],cats[:,1]
x = F.dropout(torch.cat([self.u(users),self.m(movies)], dim=1), 0.75)
x = F.dropout(F.relu(self.lin1(x)), 0.75)
return F.sigmoid(self.lin2(x)) * (max_ratingmin_rating+1) + min_rating0.5
wd=5e4
model = EmbeddingNet(n_users, n_movies).cuda()
opt = optim.SGD(model.parameters(), 1e2, weight_decay=wd, momentum=0.9)
We will be using MSE loss as the optimizing metric
fit(model, data, 3, opt, F.mse_loss)
[ 0. 1.0879 1.10568]
[ 1. 0.81337 0.82665]
[ 2. 0.80449 0.79857]
Let’s explore the learning rate
set_lrs(opt, 1e3)
fit(model, data, 3, opt, F.mse_loss)
[ 0. 0.68968 0.79054]
[ 1. 0.71873 0.78805]
[ 2. 0.70101 0.78719]
Neural Network Approach
 We can add dropouts
 we can use embeddings of different sizes
 we can add more hidden layers with more nodes
 we can add different amounts of regularization
Talk about the training loop
What’s happening inside?
opt = optim.SGD(model.parameters(), 1e2, weight_decay=wd, momentum=0.9)
To the Excel!
First create some data
Can we learn the slope and intercept?
Lets look at the error equation (err^2=1,849)
Increasing the intercept (1.5 up from 1) changes the error (err^2=419)
How can we guess where we should go next? Finite Differencing
A similar related term is backpropagation (the figure on the right)
Caveat: there’s a problem with finite differencing at highdimensionality
1,000,000 > transformation > 100k
When you have a high dimensional space, your gradient becomes a huge matrix. The amount of calculation computation that you need is very high. Takes up a lot of memory, a lot of time.
Look up:
 Jacobian
 Hessian
More efficient: find the derivatives analytically instead of finite differencing.
Remember the chain rule from calculus (quick aside on how the chain rule works)
Explaining the manual calculation of the differentials, to calculate a new a
and new b
(orange), then run all back again.
Momentum: addressing how fast we come to a good solution
 Looking at SGD
de/da
, we see that it is positive and negative at random.  The momentum idea is controlled by beta coefficients, will keep the differential going in the same direction, but a bit faster; will reduce the number of iterations.
 Adam is much much faster, but the answers are not as good as SGD with momentum.
 Recently, a newer version is out: AdamW that fixes weight decay issues. Ideally this will be fixed going forward.
About Adam Optimizer: even Faster!
Equals to our previous value of b
, times learning rate divided by sqrt()
.

cellJ8
= linear interpolation of derivative and the previous direction 
cellL8
= linear interpolation of derivative squared + derivative squared from last step  this is also known as exponential weighted moving average
Some intuition on this (reminder, in the denominator)
 squared derivative is always positive
 if there’s a big change, the squared derivative will be huge
 so if there’s a big change, we divide the learning rate by a big number, (slow down)
 if there’s minimal change ( we will be more aggressive in choosing a new learning rate)
 adaptive learning rate  since the rate isn’t constant, but dependent the model’s optimization movements
Similar idea implemented in fastai libary
avg_loss = avg_loss * avg_mom + loss * (1avg_mom)
From the fit function:
def fit(model, data, epochs, opt, crit, metrics=None, callbacks=None, **kwargs):
""" Fits a model
Arguments:
model (model): any pytorch module
net = to_gpu(net)
data (ModelData): see ModelData class and subclasses
opt: optimizer. Example: opt=optim.Adam(net.parameters())
epochs(int): number of epochs
crit: loss function to optimize. Example: F.cross_entropy
"""
stepper = Stepper(model, opt, crit, **kwargs)
metrics = metrics or []
callbacks = callbacks or []
avg_mom=0.98
batch_num,avg_loss=0,0.
for epoch in tnrange(epochs, desc='Epoch'):
stepper.reset(True)
t = tqdm(iter(data.trn_dl), leave=False, total=len(data.trn_dl))
for (*x,y) in t:
batch_num += 1
loss = stepper.step(V(x),V(y))
avg_loss = avg_loss * avg_mom + loss * (1avg_mom) #<
debias_loss = avg_loss / (1  avg_mom**batch_num)
t.set_postfix(loss=debias_loss)
stop=False
for cb in callbacks: stop = stop or cb.on_batch_end(debias_loss)
if stop: return
vals = validate(stepper, data.val_dl, metrics)
print(np.round([epoch, debias_loss] + vals, 6))
stop=False
for cb in callbacks: stop = stop or cb.on_epoch_end(vals)
if stop: break