Hi! Iām sorry if this is a n00b mistake. Iām using @lesscomfortableās Spanish LM, which he so graciously has a GDrive link to on the linked GitHub repo. However, in general, is something like the fwd_wt103.h5 model useful without the corresponding itos_wt103.pkl?
That is, without mapping the classification taskās vocab to the LMās vocab, would we get any benefit?
I think you are right Sam. Iāll upload the itos file tomorrow so people can use it with the model. Iāll also answer your other Github question tomorrow.
Sorry, I know this is very late, but I had this same problemā¦ it seems that, regardless of what language you are working on, fastai requires that language and English. So just !python3 -m spacy download en ā¦ and that will fix it.
With the German Language Model by @t-v i tried to classifiy emails as part of a project. After fitting the last layer and running lr_find with the default fast.ai code I get following plot:
I am looking for contributors and help take this further, specifically: experiments to compare ULimFit against other classical and Deep Learning based Text Classification approaches.
I did try to find some state of the art, but I seemed really hard - either the dataset had language quite different from Wikipedia (my impression is that Twitter datasets seem to contain a lot of colloquial terms at least for the sb10k corpus referenced above) - or the benchmark wasnāt clear to me. I donāt know how ULMFiT does on Germeval-2017 linked above, it might be good to test that . @rother or @MatthiasBachfischer might now something more.
There is a typo in the source code for some versions and I have encoutered the same issue. Quick fix is to either call get_rnn_classifer or to define a new function with get_rnn_classifier = get_rnn_classifer.
GermEval 2018 has some pretty well-suited tasks for ULMFiT : Classification and Fine Grain Classification. In case you arenāt taking part in the competition already, we can train the ULMFiT with sentence piece on the competition data and we will be able to compare the results on September 21 (the workshop day).
If you took part in the competition and won, can you share your paper or provide an appropriate citation?
We won 3rd task in poleval 2018 using ULMFiT with SentencePiece as the tokenization, unfortunately, the task was just about creating language model so we couldnāt use the transfer learning. Iām looking for an example where Sentence Piece + ULMFiT achieve SOTA in down-stream tasks to justify our claims in the paper.
If you take part in any competition with your LMās one thing that helped us the most was to try many different parameters on a very small corpus (10M of tokens) thanks to this we could check 53 combinations in just under a day.
Iāve been training a LM on clinical/medical text using the MIMIC-III database and things have been going really well. The initial model I completed today (~13 hours of training time) had a perplexity of ~15 on the validation set, with an average accuracy of 60% in predicting the next word.
The initial model is a word-level model that uses the tokenization methods from the course, this will be my base that Iāll use to compare different tokenization methods/hyper-parameters against.
The initial results seems too good to be true to me, so Iāll be digging into it a bit more to see if thereās some area where Iām allowing for information leakage, or if itās gotten really good at predicting nonsense (for example thereās a lot of upper case in my corpus, so I wonder if itās gotten really good at predicting the uppercase token). Iāll need to do some more research as well to see if thereās published papers that I can compare results against.
All in all itās pretty amazing how quickly Iāve been able to set this up and get things running, thanks to everyone in this thread for sharing their work and thoughts. Iām writing up a blog post about what Iām currently doing and will share soon as well.
We have an entry for Germeval (binary task only) but Iām fairly confident that it is not that great. Unfortunately I saw the competition late and had a very heavy workload towards the end that clashed a bit with doing more. Additionally there were some technical difficulties towards the end (heatwave in Germany + computers that crunch for 3-4 days = bad combination). We deliberately kept it very vanilla ULMFiT so I just used a 50k token Wiki German LM, about 300k self collected unlabeled Tweets and just the provided training data. No ensembling. The LM and the Twitter model are pretty decent I think (<28 perplexity and <18 perplexity respectively). The classifier eventually converged (I underestimated this step) and we got an F1 of about 0.8 on the validation set which Iād been very happy with but a rather disappointing score for the test set. Iāll discuss the final results after the event (itās this weekend). If anyone else from these forums attends, shoot me a PM and letās meet/talk
Even with the very hectic finish Iād do it again. Very many lessons learned. Iām confident that the results can be improved a good bit and have some ideas but little time
Letās clean up and get ULMFiT working on our languages
Jeremy gave us an excellent opportunity to deliver very tangible results and learn along the way. But It is up to us to get our selves together and produce working models.
I know that ULMFiT is a beast (sometimes), you need tones of memory and it taking a full day of worming your room just to see that the language model isnāt as good as you wanted. I get it, but it is how deep learning usually feels if it was easy there wouldnāt be any fun in doing this.
But we are so close. Letās get it done!
How about a multiple self-support groups?
I mean a chat where are ppl that work on the same language model. People that care that your model got perplexity of 60, and they understand if that is good or bad. And can offer you an emoji or an animated gif.
A support group == a thread for each language.
If you are in, vote below to join a language group and start training.
The first person that votes should create a thread and link it to the first post above (you have 3 votes):