Source: Natural Language Processing with PyTorch by Delip Rao et al.
Predictions: y_hat = model(x) , here we are using own model.
Loss function: loss_func(y_hat, y). In addition to that we are also adding it with w2*wd
Gradients: parameter.sub_(learning_rate * gradient), performing an inplace subtraction on parameters with product(learning_rate, gradient). But since our model has multiple parameters (weights, biases), we are looping through them using PyTorch parameters.
Extras:
Weight Decay:
a) w2: using each parameter, we are calculating the sum of squared weights, w2 , for p in model.parameters(): w2 += (p**2).sum()
b) wd: a constant (1e-5)
multiply w2 and wd & add to regular loss_func
Combined
We are going to calculate the loss for each minibatch by calling update(x,y,lr) on them. losses = [update(x,y,lr) for x,y in data.train_dl]
.item() turns into a python number in order to plot & see them visually.
def update(x, y, learning_rate):
wd = 1e-5
#prediction
y_hat = model(x)
w2 = 0.
#sum of squared weights
for p in model.parameters():
w2 = w2 + (p**2).sum()
# regular loss
loss = loss_func(y_hat, y) + w2*wd
# updates the gradients in the model ie parameters
loss.backward()
# instruct pytorch not to record these actions for the next gradient calculation
with torch.no_grad():
for p in model.parameters():
#gradients
p.sub_(learning_rate * p.grad)
p.grad.zero_()
return loss.item()
We are deciding timing for the next meetup, please write your suggestions in the comments or vote in the slack The meetup will be dedicated to Lesson 4
Curious general question: how do you guys keep track of changes in your model. For instance, when you change parameters or change resnet#. Do you guys save the graph in word document, use a software to compare different runs or what do you guys do?
Personally, i had been trying to use Wandb to compare different models but still trying to learn how to use different features. For instance, if i wanted for it to grab a graph of a certain run and the others to compare how its changing.
Hi hi, just a reminder, the meetup is today at 4PM GMT ( 8AM PST = 9:30PM IST = 11AM EST) It is dedicated to NLP and based on Lesson 4. Click to the Zoom link when it is a time.
This worked when I tried few months back, not sure if this still works but you can try this hack to automatically reconnect colab notebook. https://stackoverflow.com/a/58275370/5951165
@Tendo also shared Colab for Amazon Review Sentiment Analysis
Questions
What does np.random.seed(2) do?
This post & this one addresses them
The ImageDataBunch creates a validation set randomly each time the code block is run. To maintain a certain degree of reproducibility the np.random.seed() method is built-in within the fastai library.
How to stop Google Colab from disconnecting? @vijayabhaskar shared this in the previous comment
Challenges
Many participants shared the common challenges that they face. This deserves a separate post (will summarize this & possible options soon)
Hi people!!! As part of this study group, we are starting an algorithms meetup to hone our expertise in using data structures and algorithms, which can be useful for interviews as well.
Preparing for Leet-code styled coding interviews can be a very challenging task because the material is scattered and finding the perfect explanation for the problem can take time. I, along with a friend prepared for these interviews and I intend to cover some patterns that we learnt, (related to data-structures and algorithms) that were useful to us. We both got a new job after weeks of preparation and iteratively figuring out how not to fail. Please note that I will be just sharing my experience and by no means am I an expert (yet ). I hope my experience will help others in solving such coding problems and nailing that interview!!!
People who are interested can join the slack for our study-group using the link in the first post of this thread. (We would be using the #coding_interview_prep channel for this specific purpose)
We will focus on Lesson 4!