In the IMDB Lecture we used the embedding created by training a language model on wiki text dataset and used transfer learning to adjust the embedding by training a Language Model on IMDB dataset. So my question is instead of training a language model again on IMDB dataset can’t we use the embedding directly and modify our classification model by including a fully connecting layer before the RNN layer (In a way this layer should also learn a mapping similar to new embedding learned from Language model). So is my thinking right? Or I’m very naive and missing some advantage that retraining a Langauge model provides?
Shubham - we adjust the embeddings learned on the wiki103 corpus because there may be
a) novel words in our new dataset, or
b) a different distribution of words compared to the source, or
c) slightly different meanings to some words in the new corpus
You can certainly use the embeddings directly, but you may not get the same performance. That being said, I haven’t done the legwork to actually research out the difference. If you do, would you mind posting your results?