Lesson 5: Create his own Linear Layer

Part 1 (2019)
1

Hi guys,

During the fasi.ai course Lesson 5 https://youtu.be/CJKnDu2dxOE at 1:30:30 he suggest to try to create his own Linear cald My_Linear for example.
To better understand how fast.ai work I tried.

class My_layer(nn.Module):
    __constants__ = ['bias']
    
    def __init__(self, in_features, out_features,bias=True):
        super().__init__()
        self.in_features = in_features        
        self.out_features = out_features
        self.weight = Parameter(torch.Tensor(out_features, in_features))
        self.bias = Parameter(torch.Tensor(out_features))
        self.reset_parameters()
        
    @weak_script_method    
    def forward(self, input):
        y_hat = input@self.weight.t()
        y_hat = y_hat+self.bias
        return y_hat
    
    
    def reset_parameters(self):
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in)
            init.uniform_(self.bias, -bound, bound)

class My_layer(nn.Module):
    __constants__ = ['bias']
    
    def __init__(self, in_features, out_features,bias=True):
        super().__init__()
        self.in_features = in_features        
        self.out_features = out_features
        self.weight = Parameter(torch.Tensor(out_features, in_features))
        self.bias = Parameter(torch.Tensor(out_features))
        self.reset_parameters()
        
    @weak_script_method    
    def forward(self, input):
        y_hat = input@self.weight.t()
        y_hat = y_hat+self.bias
        return y_hat
    
    
    def reset_parameters(self):
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in)
            init.uniform_(self.bias, -bound, bound)
model = Mnist_Logistic().cuda()

def update(x,y,lr):
    wd = 1e-5
    y_hat = model(x)
    # weight decay
    w2 = 0.
    for p in model.parameters(): w2 += (p**2).sum()
    # add to regular loss
    loss = loss_func(y_hat, y) + w2*wd
    loss.backward()
    with torch.no_grad():
        for p in model.parameters():
            p.sub_(lr * p.grad)
            p.grad.zero_()
    return loss.item()

losses = [update(x,y,lr) for x,y in data.train_dl]

And I have this error

Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 677, in __del__
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 677, in __del__
    self._shutdown_workers()
    self._shutdown_workers()
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 659, in _shutdown_workers
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 659, in _shutdown_workers
    w.join()
  File "/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 138, in join
    w.join()
  File "/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 138, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process

What I understood it is that the weight are normalize during the init part and during the training (forward) the weight are updated.
I had an error before (tensor size mismatch) but it was because I forgot to transport self.weight.
But then this error happen but I don’t understand what I missed.

Thanks for your help.

1 Like
2

It seems like this is probably an error with PyTorch and that the actual code might be fine.

See here and here.

1 Like
3

Yeah I saw this post and it look like I won’t be able to resolve that.
Thanks for your help :slight_smile:

4

Hi, in lesson 5 I’m not able to download the dataset ML-100k and I’m using google colab.
How can i get the dataset?
@ilovescience

Capture
Capture.PNG1297×606 37.2 KB

5

I think you are already downloaded the dataset. The problem is you are reading from a wrong place.

Check your system path with pwd.

Normally in Colab I change my system path by:

import os
os.chdir('/content/gdrive/My Drive/Colab Notebooks')

From there it is easier to check the file.

1 Like
6

@AjayStark the dowloaded file is not downloaded in the right directory that is Config.data_path(). Use this in line 13 of your code in the above screenshot:

path = Config.data_path()
! wget -P {path} http://grouplens.org/datasets/movielens/ml-100k.zip

After this u can carry after this.

1 Like
7

@dhoa Hi, it worked. Thanks for the response :slight_smile:

8

Hi @umaidzz it works, but when i give the names(user, item,ratinig,timestamp), it says user is not defined.

Capturesc
Capturesc.PNG1347×602 34.9 KB

9

@AjayStark the elements: user and item in [user, item, ‘rating’, ‘timestamp’] are not in single quotes which passes them as variables instead of string which have not been defined. Replace:

name = [ user, item, 'rating', 'timestamp']

with
name = [ 'user', 'item', 'rating', 'timestamp']

or add this line at the top of this cell:
user, item = ‘user’, ‘item’

1 Like
10

I restarted runtime and now i’m not even able to unzip
@umaidzz

gsgesg
gsgesg.PNG1340×615 35.5 KB

11

@AjayStark If u restart your kernel, you don’t need to unzip the folder again because the data was downloaded once and then u unzipped it before. So now u need to skip these 2 steps as they are not required because data is already present

1 Like
12

@AjayStark If you want to understand whats happening in these lines. The lines with exclamation mark at the start indicate that it is a terminal command so in your terminal of jupyter instance go to the data directory by running

cd .fastai/data

Then print the directory with

ls

You will find your ml-100k.zip file there then use unzip

unzip ml-100k

A directory by the name of ml-100k will be created

These steps are already done by you once so the directory has already been created by you the first time you ran the commands

1 Like
13

@umaidzz Oh, i understand.
Thanks for the detailed explanation :slight_smile:

1 Like
14

Hi,
In lesson 5, movielens, how to create dataset using datablock api?
I tried this:

np.random.seed(42)
data=(CollabList.from_df(ratings_movie)
.split_by_rand_pct(0.2)
)
But how to set item_name=title?
@umaidzz

15

just do .from_df(… item_name = 'title …)

1 Like
16

@umaidzz yeah got it, thanks
And I’m trying to write nn.Linear(from below) from scratch.

class mnnist_logistic(nn.Module):
def init(self):
super().init()
self.lin1= nn.Linear(784,10, bias=True)

def forward(self, xb):
return self.lin1(xb)

In this nn.Linear performs: x@a+b and I have x, how do it get a and b (i.e the embedding and bias)?

17

You create a a tensor and initialize them with ones or zeros by using the function torch.zeros or torch.ones()

1 Like
18

oh ok and the nn.Linear does the same thing?

19

Yes, you can look up the pytorch docs u will find the same thing

1 Like