Pre-trained text encoder

[LessW2020 (Less)]

@heye0507

Also I am trying to figure out how to load different encoders, such as BERT/GPT-2 into fastai. Any input is highly appreciated!

I made a new thread on this very topic and linked in a good article showing how to do it.

Moving BERT into FastAI - good article

I’m planning to do some work with BERT but wanted to use the FastAI library…I found this article about how to do that: I started a new thread as I saw other people asking about doing the same. I haven’t worked through the above yet but plan to do so tomorrow and will post if I find any other tips/tricks involved (and please do the same if you integrate). Hopefully we can establish an optimal process for integrating BERT, RoBERTa, etc.