Word Embeddings

Hi
I was wondering how to get contextual word representations like ELMO from ULMFIT . I have the fine tuned language model and need to get the word embeddings to be able to use somewhere . Any code should be helpful

Thank you

3 Likes

Iā€™m curious about this as well ā€“ did you ever figure it out?

Hi
Zache

You can use the final layer of the LSTM encoder . The embedding size is 400

load_model(encoder, './models/lm1_enc.h5')
encoder.reset()
encoder.eval()
encoder.cuda(0)
model = BiLSTMModel(embedding_dim=EMBEDDING_DIM, hidden_dim=HIDDEN_DIM,  label_size=3,\
                                              use_gpu=True, batch_size=32)
model.cuda()


for i,batch in enumerate(data):
                        raw_outputs,outputs = encoder.forward(Variable(batch[0]))

outputs is a tensor sentencelen* batch_size*embeddingdimension

you have to use this ahead for your BILSTM model

Hope this helps

2 Likes

Thank you!!