Sentence similarity

(Brian) #21

@jeremy I’ve been attempting to use the pre-trained LM from lesson 10 to create sentence vectors. I’d like to use the vectors to create a semantic search system.
My first attempt at using pooled hidden states as vectors ( described here ) showed that semantically different sentences weren’t appreciably different from semantically similar ones. Further attempts to build a classifier from the LM to predict entailment yielded similar results. The classifier is a Siamese Network available here


  1. Should the pooled hidden states of a LM produce vectors suitable for determining sentence similarity? In other words, would you expect 2 semantically similar sentence to have a greater cosine similarity than 2 unrelated sentences?
  2. I’m not sure how to proceed. Does this look like a reasonable approach? What do you do when you get stuck on a problem like this?
  3. Am I missing something obvious?

Any insight is greatly appreciated.

(Jeremy Howard (Admin)) #22

ULMFit is all about fine-tuning. I wouldn’t expect it to work without that step. I would expect it to work well for semantic similarity if you fine tune a siamese network on a ULMFit encoder.

(Brian) #23

Thanks for your input!

(Brian) #24

Update on my progress:
I made 2 changes that have boosted my performance from 40% to 50%.
The first was to sort my sentences by length.
The other was that I switched to the MultiBatchRNN encoder.

50% is still a very poor result, so I’m going to dig in further to the InferSent code to what might be different.

The other thing I did was to validate my loader and model code with the original IMDB task.
I was able to get good results, but not as good.

Update: I’ve gotten 61% accuracy now. Better but not great. Infersent gets an accuracy of 84.5% on SNLI.

(Jeremy Howard (Admin)) #25

You’re making quite progress!

(Brian) #26

Update: I changed my vector concatenation to the way that InferSent does it. So my forward pass now looks like this:

def forward(self, in1, in2):
        u = self.encode(in1)
        v = self.encode(in2)
        features =, v, torch.abs(u-v), u*v), 1)
        out = self.linear(features)
        return out 

This has improved my accuracy to about 71%

(armin behjati) #27

I read your code in your ULMFiT_Classify notebook and it seemed that you don’t tokenize your input when making the Dataset . sorry if I’m wrong .

(Brian) #28

Tokenizing happens in a different notebook. I wanted to split-up tokenizing, pertaining and classification to make the notebooks clearer.

(Avin Regmi) #29

When I’m running your ULMFit_classify after running other two notebooks it’s giving me an error. Says “ValueError: optimizing a parameter that doesn’t require gradients”. I have been stuck here for the past few hours!!

(Brian) #30

Hey sorry you got stuck!
Looks like you are using an earlier versions of PyTorch. FastAi defaults to 0.31 but I’m using 0.4.1.

Please upgrade and try again.

(Brian) #31

Did upgrading to 0.4.1 work fo you?

(Avin Regmi) #32

Hello Brian, I got it to work but I’m not able to train it. When I’m running the fit function with multiple epochs I’m running out of GPU memory. I’m using Google Compute Engine with tesla 11gb memory. Which GPU did you use? Also if its possible can you please upload your pre-trained model? That would be a really good help for me. I’ve attached a picture where I run out of memory. After running this cell, it gives me CUDA memory error. I’ve also tried reducing the batch size.


(Brian) #33

I’m using a GTX 1080Ti. It’s also has 11GB of memory. You can try reducing the BPTT parameter as well as the batch size.

I can upload my retrained model, but it’ll take a while. I’ve got a slow upload speed.

(Avin Regmi) #34

Thanks, Brian. I really appreciate your help. Are you uploading to github or another source?

(Brian) #35

@avinregmi Here is my pre-trained SNLI language model. Hope it helps:

(Avin Regmi) #36

Thanks, Brian!!!

(Avin Regmi) #37

Hey Brian, I only found one file for the model which is I’m looking for the file that you trained. When I try to load the language model you gave me as siamese_model it gives me an error. I believe there suppose to be another file for siamese model.

Thanks for your help again Brian!

(Brian) #38

(Avin Regmi) #39

Hey Brian, so i updated to the version or torch you’re using by $ pip install ‘torch==0.4.1’ and now i’m getting this error. I’m using the instruction from this link to install fastai Fastai v0.7 install issues thread . After creating a virtual environment, i’m installing torch 0.4.1 as the default fastai installation is 0.31. But after running ULMFIT_classify notebook i’m getting this error.

btw, this is the first cell in the notebook where all the imports are happening. Which version of python are you using. I’m using 3.6.6 and I believe that might be the issue. When I made a virtual environment with python 3.7 I’m getting another error. Please help!!!

(Stas Bekman) #40

FYI: fastai 0.7 requires pytorch<0.4

it appears you’re using fastai 1.0.x, if that’s what you want please follow the instructions completely: