A simple request

[cedric (Cedric Chee)]

Hi Esteban.

@Esteban

it sounds like ULMFit hasn’t been talked about that much in the class so far. I will just look at previously published material.

Yes, Jeremy didn’t talk much about that, I think because it will be covered in much deeper in the advanced part of the course (maybe the next version of part 2).

I am one of the contributor to the Language Model Zoo for Malay language:

ULMFiT for Malay Language Project

Hey, I am pleased to introduce you to our State-of-the-Art Language Modeling and text classification in Malay language with perplexity of 29.30245 on Malay Wikipedia and 77.5% accuracy on DevCon’s Malaya dataset. Summary: the benchmark shows that using ULMFiT for text classification currently outperforms models built using classical machine learning or other neural networks. Benchmark Performance and result of various models for LM and sentiment analysis: Type Model Dataset Metric Value …

You can learn more about ULMFiT in lesson 4 (basic) and lesson 10 (deep dive into AWD-LSTM network, language modelling, fine-tuning, and more).

If you prefer to read, you can check my lesson 10 notes. The exact part where Jeremy started to talked about:

• NLP
• fastai.text module
• IMDB language modelling with fastai.text API
• pre-training on Wikipedia dataset
• fine-tuning
• … and so on and so forth

Although my work is done for ULMFiT using fastai v0.7.0 (the older version), it does not ends there. If you prefer to use fastai v1 for your work, here’s a summary of what’s going on (not trying to be accurate). You can join and follow along.

Many of us are still working to get our ULMFiT language notebooks updated to fastai v1:

• Charin (Thai): ULMFit - Thai
• NandoBr & monilouise (Portuguese): Language Model Zoo 🦍 - #328 by NandoBr

We have ironed out many issues while trying to adapt to fastai v1.

Sebastian Ruder, Piotr, Charin, and a few of us are working:

• to get ULMFiT updated to use fastai v1
• training a new ULMFiT using AWD-QRNN (trying to show this is more efficient)

You can see the dev progress in this thread:

ULMFit - Portuguese

Great! We are getting ulmfit updated to version 1 here: Multilingual ULMFiT , we will have a separate repo, and we will kill the memory issues. I think to compare with SOTA you can still use the old version, so we have the results and we can rewrite them to v1 once the issues are fixed. Awesome application thank you for sharing I hope we would have such proactive guys working in federal offices Poland! Report this as an issue in github fastai and let’s try to fix this. I h…

We believe we can get a better result if we use fastai v1.

In addition, with Google’s multilingual BERT released, I think it will be interesting to compare ULMFiT against BERT.