ULMFiT English weights?

So I was looking around in the forum and I was able to find ULMFiT pretrained weights for all major languages posted by fast.ai community, but I wasn’t able to find the english weights?

Where can I find them? Are they available in the fastai python library?

There are pretrained weights based on the (English) wikitext-103 dataset.

This is covered in lesson3-imdb.ipynb; or checkout the documentation: https://docs.fast.ai/text.html.

1 Like

But not for the language model and the classifier, I tried training them on my own and after a good bunch of hours the results were far from the ones shown in the lesson3. So do you know where to find the encoder model and the classifier model that we save in the lesson3?

Also the link you provided only shows the baseline model not the finetuned one, which was shown in the lesson 3 video.

This isn’t part of the model zoo; the language model and classifier is specific to the dataset and task you’re working with. You do this finetuning yourself, which let’s you tackle problems and data no one has worked on before.

This sounds like the real problem. You should try to understand why the results were far from the ones shown in lesson 3.

If you run through that notebook you can generate the models yourself. They are in a subdirectory (models/ I think). Another student could provide these to you, but that misses the point; if you can’t finetune your own language models and train your own classifiers you won’t be able to solve NLP problems.

What’s the problem you’re trying to solve? Are you trying to classify the sentiment of movie reviews?

I’m applying sentiment analysis on sentiment140 dataset (twitter dataset), because I’m not targeting a specific type of corpus, instead I’m trying to make a general purpose solution.

I’d say Twitter data is a specific type of corpus; but it definitely would cover a wide range of topics and be useful for other short pieces of text.

For this kind of thing I’d think the ULMFiT approach should still work; using a pretrained IMDB classifier isn’t likely to give you the best solution (especially since that only had positive and negative and the sentiment140 dataset also has a neutral class).

You said the results were far from the ones in lesson3. In what ways? Was the language model producing bad sentences? Was just the classifier really bad? Maybe something went wrong in training that could be fixed.

If you post some more details someone in the forum might be able to help you find what’s going wrong.

The training set of sentiment140 says that it contains neutrals, but when you check there are only positive and negative classes.

I also think the training is at fault in this one, I’ll try again.

Thanks :smiley:

1 Like

To reply to the main question of this forum, language model weights should be available from the pretrained English model, of the language_model_learner object.