Hey. My dataset is a mixture of French and English and I have a classification problem. Can you give me some advice on using Ulmfit? Should I train a new LM on mixed French and English wiki? Thanks
Is the text within a document in your dataset multi-language or is each document in only one of multiple languages? If the latter is the case, you can use something like the langdetect Python library. Run
detect_language() to find the probabilities for each language and remove the document if the probability for your language is below a given threshold.
In my case I have a dataset of book reviews that are mostly in Dutch. Some reviewers have translated the Dutch review into other languages, so I use langdetect to remove these multi-language documents from the dataset.
I hope this helps!
Hi, thanks for the response.
I tried langdetect but as my dataset consists from descriptions which are not very long, it found ‘wrong’ languages with very high probability.
So having the fact that I cant detect language, I am thinking of training a new model from scratch based on French and English wiki and would like to get an advise.
you would have to fill all the hyper params then. It is easier to train the LM with LMHyperParams then load it using
CLSHyperParams.from_lm(dataset_path, base_lm_path). So in your case:
exp = LMHyperParams(dataset_path='data/wiki/ru-100/', qrnn=False, tokenizer='v', lang='ru', name='russian')
exp = CLSHyperParams.from_lm('data/my_class_task_data', 'data/wiki/ru-100/models/v70k/lstm_russian.m', name='russian') # get the exact name of your model from execution of LMHyperParams
This part isn’t ready yet you would have to modify this file: https://github.com/n-waves/ulmfit-multilingual/blob/master/ulmfit/train_clas.py#L102-L107
Can you suggest something and make a PR, I would love to incorporate Russion into the ulmfit-multilingual.
I would based it on the
Russian is a bit like Polish meaning that both languages have rich morphology so I would use sentencepiece tokenizer with 25k tokens.
Here is how I would train russian LM:
python -m ulmfit lm --dataset-path data/wiki/ru-100 --bidir=False --qrnn=False --tokenizer=sp --name 'bs40' - train 20 ---bs=40
Having Dutch in ulmfit-multilingual would be awesome is your dataset public?
If you may want to use a merged wikipedia’s, to train this kind of LM. @eisenjulian is working on very similar task maybe he has some code to share.
You may as well think about using LASER it was released 5 days ago. https://github.com/facebookresearch/LASER
Thank you @piotr.czapla. As for now i’m working on the level of fastai abstraction to make it more simple (for me) to understand all the logic. I’m trying to work on upstream classification tasks and get near the best results (in comparison with current benchmark - http://text-machine.cs.uml.edu/projects/rusentiment/).
Next I’ll try to increase the amount of domainspecific (twiter) data for LM finetuning to see if it helps on upstream classification task.
Later I’ll experiment with different tokenizers and try to addapt train_clas.py to my data.
@ademyanchuk Perfect ! Can you create a “ULMFiT - Russian” page and put it on the top page of Language Model Zoo? @ppleskov was working on the Russian model in the past but I don’t think he managed to beat the SOTA Russian Sentiment Analysies, He was using the following benchmark: http://www.dialog-21.ru/evaluation/2016/sentiment/
Shure. I’m not currently best on this one
But i actuallty did some experiments on
and it turned out to be easy task to classify, i get near .98 f1. Check my solution twice on any bugs, but cant find something. So I believe ruSentEval is beaten.
I create thread for ULMFit - Russian ULMFiT - Russian
I just posted about a German topic classification dataset to the ‘ULMFiT German’-page. The dataset might be useful for some of you here.
I’m very happy to share the Dutch dataset and the weights of the trained language model!
However, for the former, I’m not sure whether I can publish it, since the contents are scraped. I’ve read the website’s disclaimer and there’s nothing in it that forbids it, but I’m still not quite sure about legal issues
For language model weights, do you have pointers how I can best package and describe it? I’ve never shared network weights before.
I believe that in most jurisdictions scraping is allowed, unless it is explicitly restricted by the TOS of the site.
@jeremy That sounds promising!
I still wanted to give credits to the owners of the website and its reviewers, so I’ve sent them an email to see if they’re open for this kind of publication. I’ve already tidied up my code and the dataset itself, so it’s ready for publication. I’m just waiting for a response from them now… Fingers crossed!
@sgugger: thanks for the heads-up!
@piotr.czapla: considering recent developments in the FastAI library, what’s the status of ulmfit-multilingual? I’ve used slightly edited versions of those scripts to generate a language model for Dutch. Is this still the way to go?
@benjaminvdb, Good work, and on time so we can add it to the paper that summarise multilingual ULMFiT.
Recently @mkardas made ulmfit-multlingual compatible with the fastai, we have it in a separate branch that I’m looking in to merging to master tomorrow.
ULMFit - Portuguese
ULMFiT - Japanese
Gals & Guys we are summarising the work on ulmfit-multilingual, and @sebastianruder is writing a short paper.
I’m going through the forum threads and I’m adding ppl that contributed some interesting tests to a private thread, but there is a chance I’ve missed something. If you feel you should be added please let me know.
You can check if you are in the thread by following this link. If I missed you please let me know on private message or thread @ mention me in your language that shows the results.
The paper has a deadline soon (end of Feb). If you would like your results to be added we need to have a pull request by next week that includes:
- the code to download your dataset
- the “.md” with the results
- and links to pretrained LM weights.
So we can run the classification task to double check the results are reproducible without need to train the full LM. If you happen to contribute to the language covered by MLDoc the LM’s won’t be necessery as we will have both lstm and qrnn based one.
Just to chip in, everyone who contributed a dataset and achieved good results that we can reproduce on the data will be included on the paper. Please provide additionally the following information in the PR so that it will be easier to attribute the work:
- your first name and last name;
- your preferred affiliation;
- your email address.
Thanks for your efforts!
Finalizing results for Thai to use fastai 1.0.38 and above (previous results and datasets are based on 1.0.22). Do you have a hard deadline for the pull request? @sebastianruder
Edit: sorry I think I saw this 12 days too late
I’m DS @ KPN, NLP practitioner and leader of mlcourse.ai.
Currently doing SSL in parallel with English (amazon product reviews) and Dutch texts. Have tried a lot of models and approaches, and currently ULMFiT works best for reviews in English.
@benjaminvdb thanks a lot for sharing the pretrained Dutch model!
I only regret that found this thread so late. If any contribution from my side needed - I’m open.