How can I train a LM with a smaller vocab on the pretrained Wiki LM?

Part 1 (2019)
1

Is it possible to train a custom Language Model with a smaller vocab (ie 10.000 words) on your pretrained wiki of 60.000 words. When I try to load the weights I get this error.

RuntimeError: Error(s) in loading state_dict for SequentialRNN:
	size mismatch for 0.encoder.weight: copying a param with shape torch.Size([60002, 400]) from checkpoint, the shape in current model is torch.Size([10002, 400]).
	size mismatch for 0.encoder_dp.emb.weight: copying a param with shape torch.Size([60002, 400]) from checkpoint, the shape in current model is torch.Size([10002, 400]).
	size mismatch for 1.decoder.weight: copying a param with shape torch.Size([60002, 400]) from checkpoint, the shape in current model is torch.Size([10002, 400]).
	size mismatch for 1.decoder.bias: copying a param with shape torch.Size([60002]) from checkpoint, the shape in current model is torch.Size([10002]).
2

checkout the line where you formulate a TextClasDataBunch. There you can pass a vocab kwarg and for example assign: data_lm.train_ds.vocab if these are your variables. I hope I understood your problem, happy coding :slight_smile:

3

Hi, were you able to fix the issue? I’m stuck with the same problem. A lot of people came across the same issue but it seems nobody has a definitive answer for it.