Lesson 4 In-Class Discussion ✅

Part 1 (2019)
103

There is bias in language models. I’ve talked about it some here and here (including some approaches for dealing with it).

13 Likes
104

It exists

1 Like
105

Can Jeremy mention what exact GPU he used for the IMBD notebook, especially considering the fact that many people run out of GPU memory running it using P4?

1 Like
106

So is the wikitext103 base model somewhat superior to other models such as reddit because wikitext contains more english vocabulary ?

3 Likes
107

i believe it was a 1080Ti with 12GB

108

Is the idea of pre-trained language model similar to BERT that i came across while looking for language model.

7 Likes
111

What if you want to keep the unknown words (rather than replacing them with xxx)? As long as its used more than 1x, will it be kept?

5 Likes
112

The core idea is that even if wikitext is not that similar to the corpus you are interested in, you can still pre-train with wikitext, and then fine tune using the corpus you’re interested in.

3 Likes
113

Is fastai library available by default via Kaggle kernels? It seems they don’t allow to install custom packages into GPU kernels. There is Quora kernel-only competition so I wonder if it is possible to use the library here.

118

How to change the size of vocabulary? 6000 to 8000?

119

Can Jeremy confirm?

120

Please post these kind of questions in the Advanced Section. But yes, BERT is another example of transfer learning in NLP, using a different backbone (transformer) and solving a different task (masked language model + next sentence prediction).

3 Likes
121

Do we build the vocab again from scratch or we use the vocab from pre-trained wikitext model?

4 Likes
122

wouldn’t you be missing word if vocab is limited to a small number.?

123

Jeremy said the traditional approach is to convert everything to lower case. Then how do you predict case? Is there a separate model / technique for that?

4 Likes
124

you wouldn’t be able to use the built-in language models for that competition because external data is prohibited

125

We build it from scratch because we have new words that didn’t exist on wikipedia, then we match them.

3 Likes
126

What about other languages than English?

2 Likes
127

There is a token that specifies if the next word is caps. In test time, it works like any other token.

2 Likes
128

Already answered in this same thread.

1 Like