Running IMDB notebook under 10 minutes

Part 1 (2018) Beginner (2018)
1

Since lecture 4 I’ve struggled with a never-ending training of IMDB notebook. Fortunately, I got some pre-trained weights (thanks to @Moody and @wgpubs) thus I thought of creating this post to allow fellow students to explore the notebook since many of us have skipped it because of training time.

The modified notebook doesn’t include the required code to train learner for Language Modeling. Instead, below code is used to load pre-trained model:

learner.load(‘imdb_adam3_c1_cl10_cyc_0’)

Next, you may train model for the sentiment which is approx 70 minutes long job or skip this step and use pre-trained weights for that part as well.

m3.load_cycle(‘imdb2x’, 4)

Personally, I recommend using weights for Language modelling and training yourself the sentiment part. Using pre-trained model saves time which can be used to explore the pieces which otherwise are skipped due to ever running scripts.

Pretrained weights can be downloaded from here.

image

#ExploreMore

26 Likes
2

Well done @vikbehal. :slight_smile: I linked this post to Lesson 4 wiki. BTW, all credit to @wgpubs.

5 Likes
3

I’ve updated the IMDB file.

Updated file has information about object types, model structure, calculation/logic (for those with less knowledge about Pytorch and/or numpy), split etc.

2 Likes
4

I am getting an error while trying to run this notebook:

learner.load('Weights/imdb_adam3_c1_cl10_cyc_0')

KeyError                                  Traceback (most recent call last)
<ipython-input-33-ce0c31739924> in <module>()
----> 1 learner.load('Weights/imdb_adam3_c1_cl10_cyc_0')

~/fastai/courses/dl1/fastai/learner.py in load(self, name)
     70     def get_model_path(self, name): return os.path.join(self.models_path,name)+'.h5'
     71     def save(self, name): save_model(self.model, self.get_model_path(name))
---> 72     def load(self, name): load_model(self.model, self.get_model_path(name))
     73 
     74     def set_data(self, data): self.data_ = data

~/fastai/courses/dl1/fastai/torch_imports.py in load_model(m, p)
     24 def children(m): return m if isinstance(m, (list, tuple)) else list(m.children())
     25 def save_model(m, p): torch.save(m.state_dict(), p)
---> 26 def load_model(m, p): m.load_state_dict(torch.load(p, map_location=lambda storage, loc: storage))
     27 
     28 def load_pre(pre, f, fn):

~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    492             missing = set(own_state.keys()) - set(state_dict.keys())
    493             if len(missing) > 0:
--> 494                 raise KeyError('missing keys in state_dict: "{}"'.format(missing))
    495 
    496     def parameters(self):

KeyError: 'missing keys in state_dict: "{\'0.encoder_with_dropout.embed.weight\'}"'

Any idea what might be wrong here ?

1 Like
5

Got this error when trying to load imdb_adam3_c1_cl10_cyc_0:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/.conda/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    481                 try:
--> 482                     own_state[name].copy_(param)
    483                 except Exception:

RuntimeError: invalid argument 2: sizes do not match at /opt/conda/conda-bld/pytorch_1512387374934/work/torch/lib/THC/generic/THCTensorCopy.c:51

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-20-69866d51d4ab> in <module>()
----> 1 learner.load('imdb_adam3_c1_cl10_cyc_0')

~/fastai/courses/dl1/fastai/learner.py in load(self, name)
     62     def get_model_path(self, name): return os.path.join(self.models_path,name)+'.h5'
     63     def save(self, name): save_model(self.model, self.get_model_path(name))
---> 64     def load(self, name): load_model(self.model, self.get_model_path(name))
     65 
     66     def set_data(self, data): self.data_ = data

~/fastai/courses/dl1/fastai/torch_imports.py in load_model(m, p)
     24 def children(m): return m if isinstance(m, (list, tuple)) else list(m.children())
     25 def save_model(m, p): torch.save(m.state_dict(), p)
---> 26 def load_model(m, p): m.load_state_dict(torch.load(p, map_location=lambda storage, loc: storage))
     27 
     28 def load_pre(pre, f, fn):

~/.conda/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    485                                        'whose dimensions in the model are {} and '
    486                                        'whose dimensions in the checkpoint are {}.'
--> 487                                        .format(name, own_state[name].size(), param.size()))
    488             elif strict:
    489                 raise KeyError('unexpected key "{}" in state_dict'

RuntimeError: While copying the parameter named 0.encoder.weight, whose dimensions in the model are torch.Size([34933, 200]) and whose dimensions in the checkpoint are torch.Size([34945, 200]).
6

Had slightly different error:

RuntimeError Traceback (most recent call last)
~/anaconda2/envs/fastai1/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
481 try:
–> 482 own_state[name].copy_(param)
483 except Exception:

RuntimeError: invalid argument 2: sizes do not match at /opt/conda/conda-bld/pytorch_1512387374934/work/torch/lib/THC/generic/THCTensorCopy.c:51

During handling of the above exception, another exception occurred:

RuntimeError Traceback (most recent call last)
in ()
----> 1 learner.load(‘imdb_adam3_c1_cl10_cyc_0’)

~/fastai/courses/dl1/fastai/learner.py in load(self, name)
62 def get_model_path(self, name): return os.path.join(self.models_path,name)+’.h5’
63 def save(self, name): save_model(self.model, self.get_model_path(name))
—> 64 def load(self, name): load_model(self.model, self.get_model_path(name))
65
66 def set_data(self, data): self.data_ = data

~/fastai/courses/dl1/fastai/torch_imports.py in load_model(m, p)
24 def children(m): return m if isinstance(m, (list, tuple)) else list(m.children())
25 def save_model(m, p): torch.save(m.state_dict(), p)
—> 26 def load_model(m, p): m.load_state_dict(torch.load(p, map_location=lambda storage, loc: storage))
27
28 def load_pre(pre, f, fn):

~/anaconda2/envs/fastai1/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
485 'whose dimensions in the model are {} and '
486 ‘whose dimensions in the checkpoint are {}.’
–> 487 .format(name, own_state[name].size(), param.size()))
488 elif strict:
489 raise KeyError(‘unexpected key “{}” in state_dict’

RuntimeError: While copying the parameter named 0.encoder.weight, whose dimensions in the model are torch.Size([34933, 200]) and whose dimensions in the checkpoint are torch.Size([34945, 200]).

7

That means that your vocab is slightly different to the model that was trained. @vikbehal perhaps you could also share your torchtext field with the vocab in it?

1 Like
8

Same error. Is there a resolution to this?

9

Hey thanks for this, this is great! One question though: How have you downloaded the weights into the Paperspace machine? Wget is not working with the Chrome link…

Thanks in advance!

10

one possible way is to download the file to local machine then upload it to cloud.

1 Like
11

A brute force way is to set md.nt to 34945 using md.nt = 34945.
At least the notebook can run without running exception.
But i am not sure any side effect.

12

I tried that technique but seem to have the same issue. I’m running the notebook ‘as is’ but added setting md.nt = 34933 and m3.nt = 34933 but the dimension mismatch is present still. Is there a way to use the same method of getting the text on both (e.g., currently the first model is using the downloaded local IMDB files and the second gets them from an online location.

13

Can you be more specific? Do you mean later accessing the cloud from paperspace? I really don’t know how to access a browser after sshing into the paperspace machine.

14

Do you have a dropbox account? If so simply upload the zip folder with the pre-trained models to Dropbox and create a shareable link. Use wget and the shareable link to download the zip folder after sshing into paperspace. (You can use any cloud provider you like as long as it gives you a direct access link).

15

Unfortunately I am still getting an import error, after setting md.nt=34945

---------------------------------------------------------------------
KeyError                            Traceback (most recent call last)
<ipython-input-29-69866d51d4ab> in <module>()
----> 1 learner.load('imdb_adam3_c1_cl10_cyc_0')

~/fastai_course_playground/dl1/fastai/learner.py in load(self, name)
     62     def get_model_path(self, name): return os.path.join(self.models_path,name)+'.h5'
     63     def save(self, name): save_model(self.model, self.get_model_path(name))
---> 64     def load(self, name): load_model(self.model, self.get_model_path(name))
     65 
     66     def set_data(self, data): self.data_ = data

~/fastai_course_playground/dl1/fastai/torch_imports.py in load_model(m, p)
     24 def children(m): return m if isinstance(m, (list, tuple)) else list(m.children())
     25 def save_model(m, p): torch.save(m.state_dict(), p)
---> 26 def load_model(m, p): m.load_state_dict(torch.load(p, map_location=lambda storage, loc: storage))
     27 
     28 def load_pre(pre, f, fn):

~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    488             elif strict:
    489                 raise KeyError('unexpected key "{}" in state_dict'
--> 490                                .format(name))
    491         if strict:
    492             missing = set(own_state.keys()) - set(state_dict.keys())

KeyError: 'unexpected key "1.decoder.bias" in state_dict'

Does anyone have any idea what’s going wrong?

16

I uses google colab instead of paperspace. My solution is firstly download the file to my PC, then upload it to google drive. Hope this information is helpful.

1 Like
17

You can use CurlWget extension for chrome. It can retrieve the wget command after initiating the download command.

2 Likes
18

Have you figured it out yet? I am having the same error.

20

@jeremy Why would the vocab be different if the language model is created on the same dataset? I tried creating the model with data from files.fast.ai as well as the stanford official website since my guess was that the link was later modified in the notebook, but am getting the same result.
@vikbehal If you could share the torchtext field file, it would help out a lot!

21

I don’t really know - some different kind or preprocessing I guess…