What is an "encoder" and what is save/load_encoder() actually doing?

A few questions:

  1. What is an encoder?
  2. What part of the model are we actually saving when we run > learn.save_encoder('adam1_enc)?
  3. What is actually effected when we then load it back into our pre-trained model via m3.load_encoder('adam3_20_enc')?
3 Likes

The definition of save_encoder is:

save_model(self.model[0], self.get_model_path(name))

…so try typing self.model[0] to see what is in it. :slight_smile:

The ‘encoder’ is part of a seq2seq model, which we’ll cover in part 2 in detail. If you want to skip ahead, here’s some reading: http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

We’ll be learning a bit about it in lesson 7.

7 Likes

I’m sure Jeremy will get to this in subsequent lectures but here is the high level idea:

There are a class of deep nets which consist of 2 parts : an encoder and a decoder. You take the input and compress it into an intermediate state…then the decoder takes that and creates the output.
This is quite useful in NLP tasks like language translation where you take an English input sequence, encode into an intermediate representation (a tensor) and then decide that to a German or Hindi sequence.

In the lesson 4 notebook, we see that the imdb review text is the input and we use an encoder-decoder model that learns to predict words(write new words given a seed text). We then throw away the decoder and only save the encoder. This is quite easy in the fastai library.
We can take the encoded representation and use it for a new task (sentiment analysis) using transfer learning. Transfer learning is very common in computer vision but quite new in NLP.

Hope this clarifies.

13 Likes

So the encoder is essentially tasked with creating a mathematical representation of the language based on the task for predicting the next word.

A decoder is responsible for taking that representation and applying it to some problem (e.g., predicting the next word, understanding sentiment, etc…)

Meaning in this case …

We are creating a model that understands the English language and predicts the next word, then removing the layers that predict the next word and replacing them with layers that instead predict sentiment.

5 Likes

Precisely.

1 Like

wow…nice intution