Hi,
Is it possible to fine-tune ULMFiT on my own dataset and then extract embedding vectors from it?
Thanks
Hi,
Is it possible to fine-tune ULMFiT on my own dataset and then extract embedding vectors from it?
Thanks
Yes. That’s what it’s designed for. One of the fast.ai alumni @muellerzr has done so in one of his posts.
https://muellerzr.github.io/Suspecto/. You can check his blog. I then think since embedding layer is the first layer of the architecture. You can extract weights from them.
In the link given above, it tells you about how to fine tune on your own dataset. To extract embeddings, you could tweak the API in the following ways.
import torch
import fastai
from fastai.text import AWD_LSTM,load_data,MultiBatchEncoder, RNNLearner
lm_data_path = ‘/home/ubuntu/efs/corpus/fine_tuning_corpus/ft_corpus_250K.csv’
output_path = ‘/home/ubuntu/efs/fp16_ulmfit/vsz_100k_dp_0.75/’
bptt=70
max_len=20*70
vocab_sz=100000
drop_mult = 1
config = dict(emb_sz=400, n_hid=1152, n_layers=3, pad_token=1, qrnn=False, bidir=False, output_p=0.4,
hidden_p=0.3, input_p=0.4, embed_p=0.05, weight_p=0.5)
for k in config.keys():
if k.endswith(’_p’): config[k] *= drop_mult
ps = [config.pop(‘output_p’)]
encoder = MultiBatchEncoder(bptt, max_len, AWD_LSTM(vocab_sz, **config), pad_idx=1)
learn = RNNLearner(dl, encoder)
learn.model.reset()
learn.model = learn.model.module
learn.model.load_state_dict(torch.load(f’{output_path}models/fwd_enc.pth’, map_location=‘cuda’))
text = ‘APPLE is the world ’s greenest tech company’
batch = learn.data.one_item(text)
learn.model(batch[0])
Use hook on the classifier model made using the encoder
from fastai.callbacks.hooks import hook_output
layer = learn.model[:2][1].layers[0] # the layer from which you want to extract the output
input_ds = learn.data.one_item(text)
with hook_output(layer) as hook_forward:
preds = learn.model(input_ds[0])
print(hook_forward.stored)
executed in 3ms, finished 12:51:36 2019-12-10
In this example post, how is the csv file formatted. Is each article in one cell of the csv?
yes, each article is one row
Hey thanks. I wanted to know how would you proceed to calculate sentence embeddings . Would taking the context vector at the end of the last layer help or taking the sum of the embeddings from the embedding layer.