ULMFit - Portuguese

[piotr.czapla]

@NandoBr

Yes! Sure, let’s do it!

Great! We are getting ulmfit updated to version 1 here: Multilingual ULMFiT , we will have a separate repo, and we will kill the memory issues.
I think to compare with SOTA you can still use the old version, so we have the results and we can rewrite them to v1 once the issues are fixed.

@NandoBr

Another promising application I´m working on is a text classification model for the Federal Senate of Brazil : Every week, dozens of requests are made to one of the four team of consultants in the Legislative Consulting Department . These solicitations include requests for studies, proposals for bills, production of speeches, among others. The idea is to automatically define which team should attend the demand based on the text(usually 3 to 10 lines of text) that describes the legislative request. The preliminary result was 86% accuracy which is not fantastic , but it is good enough, considering that the human performance today is 89% accuracy.

Awesome application thank you for sharing I hope we would have such proactive guys working in federal offices Poland!

@pvcastro

I had the same problem, I did the pretraining of a Portuguese model based on wikipedia, but suffered from the lack of resources and benchmarks in Portuguese to verify the results against other models.
…
@NandoBr, I started the pretraining for the same wikipedia dump using the v1, but I ended up having memory issues, which I believe were also happening with other users. The memory consumption is increasing at every few epochs. I have a GTX 1070 with 8 Gb, and had to reduce the batch size to 16 (with 64 it was working OK on the previous version), and even doing so I ended up having out of memory error by epoch 10.

Report this as an issue in github fastai and let’s try to fix this. I had a similar issue on image classification for long training times, and if you are ok at coding try to get the issue debugged.

@pvcastro

Check this resources page from NILC, they may have a dataset for text classification to test for benchmarks. Last time I checked there was only one for sentiment analysis based on tweets.

Thank you for finding that baseline!