Saving vocab in 1.0.44

Before update 1.0.44, I used to save my vocab using the TextLMDataBunch.save method. It created a itos.pkl file that I could load to do classification. Since the last update, calling save on a TextLMDataBunch will generate a single file that contains the information needed to load the databunch but no itos.pkl . Is this intentional? And if it is, what is the best way to save my LM Vocab to load it with my classifier?

1 Like

That’s because save now works for every kind of DataBunch. There are two ways to get your vocab for your classifier:

  • manually save your vocab by pickling data_lm.vocab.itos
  • load your data for the LM with load_data then pass data_lm.vocab

I will add a save/load function for the Vocab class to make this easier.

4 Likes

Thank you for your answer. Shall I submit a PR to add the save/load function for the Vocab class?

If you want to tackle this, go ahead yes!

super…its working

Hi !
Can you provide a link to how we can save our vocab thanks to your vocab ?

Thanks in advance ! :slight_smile: