ULMFit - Portuguese

I was doing the pretraining like this, from what I could gather from the v1 docs. Not sure if it would be the best way to do it. I did end up hitting an out of memory error by the 10th epoch, as I mentioned earlier. Did you give it a try @NandoBr?

def pretrain(path, language_model_id, epochs, batch_size, use_ids=True):
    # path = Path('/fastai/data/wiki/pt')

    if use_ids:
        data_lm = TextLMDataBunch.from_id_files(path=path, train='train', valid='valid', test='test', max_vocab=60000,
                                            min_freq=2, n_labels=0, chunksize=24000, bs=batch_size)
    else:
        data_lm = TextLMDataBunch.from_csv(path=path, tokenizer=Tokenizer(lang='pt'), train='train', valid='valid',
                                       test='test', max_vocab=60000, min_freq=2, n_labels=0, chunksize=24000)

    learn = RNNLearner.language_model(data_lm, drop_mult=0.5)

    lr = float(1e-3)

    learn.fit(epochs=epochs, lr=slice(lr / 2.6, lr), wd=1e-7)
    learn.save(language_model_id)

Hi Pedro,
I didn’t have time to spend more time on it. I’m waiting for some tip on how to avoid the problem. If I were you , I would get more info about the error and post it at ULMFit main thread. Maybe somebody can help us.
Good luck and let’s keep in touch…

As discussed with @piotr.czapla, I’ll open an issue at the fastai github. I’m gathering the information needed for the issue, and will open it once I’m done.

For a quick fix simply save the best model and restart the training by loading the weights,
to do so use SaveModelCallback https://docs.fast.ai/callbacks.tracker.html

1 Like

OK, thanks!

Hi, we are trying to make summary of ulmfit efforts see: Multilingual ULMFiT
Do you have some results trained on the publicly available / official data set?

Hi @piotr.czapla ,
Yes, I can help with that. I have a few details that I’d like to discuss with you. Can I have your e-mail?

piotr.czapla@gmail.com

Ok. Thanks

Wow, really cool to find you guys around here working with portuguese datasets.

I would love to cooperate! :wink:

Hey guys!

Whats the status on this? Do we have a portuguese model already? if not, how can i help?

@piotr.czapla do you know how do i start to contribute to this? any place i can start?

Thanks and congrats for the efforts!

Hi Joao We do have a model but I suspect that it isn’t that performant as it could be as it was trained on wikipedia that has formal language. You could try to train a model that is using informal language like one used on reddit or twitter. We had good results on Polish language by pretraining on different text types. We could then test that model on some dataset and compare it to one pretrained on wikipedia.

Hello Piotr. Where can i find the notebook used to train this model? Do you guys have a sample notebook on how to train a new model? Or even how you trained for english, or this Polish model you cited? That would be awesome. I want to help but i dont know how, and really didnt want to start from scratch…
thanks and congratulations for the work being done…

Using fast.ai version 51, just follow the template below in order to train a portuguese language model:

from fastai.text import *
path = Path(“data/wiki/pt/”)
tokenizer = Tokenizer(lang=‘pt’, n_cpus=4)
lm_data = TextLMDataBunch.from_csv(path, ‘train.csv’, tokenizer=tokenizer, label_cols=None, text_cols=0, chunksize=5000, max_vocab=60000, bs=64)
learn = language_model_learner(lm_data, AWD_LSTM, pretrained=False, drop_mult=0.)
learn.lr_find()
learn.recorder.plot(suggestion=True)
learn.fit_one_cycle(10, 2e-3, moms=(0.8,0.7))
learn.save(‘pt_language_model’)
lm_data.vocab.save(path/f"models/pt_itos.pkl")

2 Likes

Hi @NandoBr, I´m working on a 60K PT-wiki LM model, and came accross your work while looking for benchmarks in portuguese. I believe that the classification results your team achieved over the TCU dataset is a great benchmark. Would you mind sharing some details, like the train-valid-test split ratios used and per class metrics (f-score/accuracy)?
Best Regards

Never mind, I´ve just found the split ratios in the notebook you shared!

I have a public school that is interested in building a chat bot to help high school children with their home work. Would love to help you guys to see if we could get something like this done. I’m thinking that the hardest part would be to get the model to recognize the math symbols, but I would be curios to see what we could obtain if we specialized the language model with about 20000 questions and answers in math.
Can I help?

Just sharing some early results from my work. Currently one can actually get up to 0.34 accuracy for Portuguese LM using the latest notebook on language modelling (https://github.com/fastai/course-nlp/blob/master/nn-vietnamese.ipynb)

I literally didn’t make much change and used only the defaults. Only change I had to made was on the nlp_util script, where I had to change the encoding to ISO-8859-1.

Also interesting to note that the loss went bonkers after the 9th epoch. The above screenshot was taken from my terminal, since I changed the notebook to a script due to the long training process. I also had the model and vocab saved every 2 epoch (via for loop) so that I my training hours wouldn’t be wasted (I’d seen the NaN beforehand).

Hi guys! I’ve trained an Portuguese model with an Wikipedia dump by using @piotr.czapla instructions, and I have arrived at an accuracy of 54.8% and 11.65 perplexity.

The repository is here: https://github.com/danilolessa/Portuguese-ULMFiT and I suggest seeing @Antti_Karlsson finnish pre-trained model repository for getting an ready and clear Jupyter notebook as for how to use it for classification tasks.

[ EDIT ] I made a correction in the notebook of my TCU classifier and got an accuracy of 97.95% higher than my previous one (see my post).


Hi @monilouise, @NandoBr e @ErickMFS.

Following the publication of the MultiFit paper on September 10 (authors: Julian Eisenschlos, Sebastian Ruder, @piotr.czapla, Marcin Kardas, Sylvain Gugger, Jeremy Howard), I wanted to verify that the MultiFiT configuration gives better results than the ULMFiT one you used in order to firstly train a Portuguese Bidirectional Language Model and then, to fine-tune a classifier of the TCU dataset (thanks for making this dataset available).

And indeed, there is a slight progression: my legal text classifier reached an accuracy of 97.39% and a F1 score of 0.9737 (less than 5mn to fine-tune the bidirectional Language Model to TCU texts and the same time required to train the bidirectional classifier).

So, I published my notebooks on github:

I also posted a post on medium explaining the motivation, the results and the configuration used.

Note: I trained my models on GCP with 1 GPU NVIDIA Tesla V100.

Bonus: I love the function show_intrinsic_attention() that allows to visualize the words that have contributed the most to the decision of the classifier. An example below with a legal text of the TCU :slight_smile:

5 Likes