Multilingual ULMFiT

piotr.czapla · April 11, 2019, 8:12am

We posted the paper and are waiting for the review I haven’t yet managed to merge all the code and update it to the newest fast.ai as we had another conference where we were using ULMFiT that we have learned about 2 weeks before the deadline so we had a thought time.

wgpubs · April 21, 2019, 8:49pm

What is the setup required to clone and get https://github.com/n-waves/ulmfit-multilingual up and running locally?

Cloned the repo but I get errors when enabling qrnn or just trying to retrain running the code in the readme:

$ LANG=en
$ python -m ulmfit lm --dataset-path data/wiki/${LANG}-100 --tokenizer='f' --nl 3 --name 'orig' --max-vocab 60000 \ 
        --lang ${LANG} --qrnn=True - train 2 --bs=50 --drop_mult=0.5  --label-smoothing-eps=0.2

Thanks

Andreas_Daiminger · April 23, 2019, 7:26am

Hey @piotr.czapla!

Seems we are on the same track. Did you manage to deploy a model trained with ULMFiT using Torch.JIT? If yes would be great if you could share your experience.

javedali.shaikh2 · April 24, 2019, 4:53pm

@piotr.czapla, @sebastianruder

dumb question - if i understood correctly this thread is about creating ULMFit model per language. Are there any plans to create single model for multiple languages (like BERT)?

maro95 · April 25, 2019, 10:21am

Hi, I’m working on ULMFiT for Croatian. I have went through this topic together with Language model zoo topic.

I have one question regarding those two topics. They have same agenda right? Agenda being creating and using ULMFiT for other (not yet implemented) languages. Or is Multilingual in the title specified for language model that can learn multiple languages (I get a different impression, hence the question)?

For creating language model I’ll use https://github.com/n-waves/ulmfit-multilingual and fastai v1 library.

cahya · May 10, 2019, 8:59am

Hi @piotr.czapla, if I use prepare_wiki.sh from your https://github.com/n-waves/ulmfit-multilingual, it will create following directories:

{LANG}-2 for small wiki with max number of tokens in each train:valid:test = 2000k:200k:200k (10:1:1)
{LANG}-100 for large wiki with max number of tokens in each train:valid:test = 98000k:200k:200k (49:1:1)
{LANG}-all for all wiki with max number of tokens in each train:valid:test = all:200k:200k (>49:1:1)

Is it on purpose that the number of tokens (at the end it is number of articles) of valid and test are the same for small, large and all wiki? In my case with indonesian wiki, I get 446632 articles as training dataset and only 256 articles in valid or test dataset for large wiki. I think the number of valid and test dataset are just too small comparing to training dataset in large wiki (could be worse in all wiki).

maro95 · May 13, 2019, 6:14pm

Hi, I need to know training time of LM, cause I need schedule GPU on server. I’m training LM for Croatian.

I mainly follow @duxan and @ademyanchuk notebooks for training LM models (they have also printed training times), so maybe you two could know most precisely. GPU is GeForce GTX 1080 (is this maybe too weak GPU for this). What GPUs you two used (or anybody that trained LM models for their respective languages)?

piotr.czapla · May 19, 2019, 11:34am

Could be but the perplexity is very poor estimation for the accuracy on downstream task it is there only ot make sure the model is being trained so I wouldn’t care too much about validation the smaller the faster the training :).

JoepJ · May 31, 2019, 9:15am

Hello,

I am doing the fastai course (2019), and I am new on this forum. I am interested in applying fastai to NLP, in the dutch language.
I looked at the pretrained models available in ulmfit-multilingual (pretrained_lm_models.zip) but it does not contain dutch.
Is there a pretrained dutch model available?

benjaminvdb · June 1, 2019, 1:05pm

@JoepJ: You could try the language model I trained on a Dutch Wikipedia corpus for a couple of days.

Let me know how it worked out for you and whether you need any help. Good luck!

JoepJ · June 1, 2019, 3:03pm

Thanks @benjaminvdb!
That saves a lot of time and effort .
I will check it out.

tanny411 · June 2, 2019, 11:28pm

I would like to contribute for Bangla Language. Can someone give me a headstart? Are there any instructions to make the wiki dataset? Would be very helpful. Thanks. Also looking forward to using sentence piece.

abyaadrafid · June 10, 2019, 4:58am

The contact person listed in the ULMFiT for Bangla seems to be inactive for over an year. Is there anyone actually working on it?
Also I found this project in the wild. Has a wikipedia Bangla corpus; didn’t get the opportunity to check it out, might be useful to you.

I’m also trying to find a way to use wikipedia data dumps. I’ll share the dataset if I manage to do something.

tanny411 · June 10, 2019, 10:04am

I havent found anyone else working on Bangla. I am currently working on it.
I have actually checked out the project you mentioned. The dataset seems small. So I was thinking of building a larger dataset.

abyaadrafid · June 10, 2019, 10:14am

Here are the data dumps: https://archive.org/search.php?query=bnwiki&and[]=year%3A"2019"

Which platform are you working on?
Both kaggle kernels are Colab time out even before they finish training on IMDB example.

tanny411 · June 10, 2019, 10:28am

i was working on colab. where can i find the imdb dataset you are refering to?

abyaadrafid · June 10, 2019, 10:31am

The one in Lesson 3 video. Colab shows me 56 hours eta on training.
This one.

tanny411 · June 10, 2019, 10:32am

Are you sure you turned on GPU?

abyaadrafid · June 10, 2019, 10:39am

Facepalm
No I didn’t. Thanks.
In my defense though, I prefer kaggle over colab.

Anyways, I’ll try to find the best database files from the dumpster and try to get it into a reasonable file format.

Time to create a ULMFiT - Bangla Thread

tanny411 · June 10, 2019, 11:05am

I have completed ulmfit already using other texts instead of wikipedia. Was wondering about the effects of wikipedia texts.