What does the encoder actually learn? 🤔

narvind2003 · March 10, 2019, 8:35pm

Hey, are you using the latest fastai version?
can you try with fastai version 0.7 please?

Serbulent · March 14, 2019, 11:48am

Thanks for answer. I am using 1.0 and I think I figure out this as;

resizeLength = len(sentenceTrimmed)
inputSentence = tensor(sentenceTrimmed).resize_(resizeLength,1)
# sentence encoding 400 dims. -1 is the last element that’s supposed to have the final encoded state
tmpEmbded = m[0](inputSentence.cuda())

I am also trying to convert your LM Evaluation notebook to fast.ai 1.0 and will share it by citing you if it is ok for you.

narvind2003 · March 14, 2019, 1:13pm

Awesome. Sure, thanks.

Serbulent · March 16, 2019, 9:15am

And it doesn’t go well of course

I can calculate distances on a pretrained model but they are wrong even for simplest words. Does anyone have an idea what could go wrong ?

github.com

serbulent/fastai_examples/blob/master/imdb-similarity-example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.datasets import *\n",
    "from pathlib import Path\n",
    "import html\n",
    "import fastai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},

This file has been truncated. show original

Serbulent · March 20, 2019, 12:26pm

One mistake that I made might be using LM encoder but even after I fixed it similarities are still not right even for words. For example word woman seems more similar to king than queen.

woman with king
0.4858151376247406
woman with man
0.8090812563896179
woman with woman
1.0
woman with queen
0.45923975110054016

harikrishnanrajeev · November 7, 2019, 4:09pm

Hi @Serbulent , were you able to make any improvements ?.

narvind2003 · April 26, 2020, 7:39am

One more paper that’s highly relevant: