Fastai Transformer and TransformerXL models

[Esteban (Esteban J Guillen)]

I see that the Transformer and TransformerXL models have been added to the github repo. Do these models fit into the ULMFit architecture or are they intended to be totally separate?

Thanks,

Esteban

[magiclantern (Seth)]

@Esteban - not sure you are still interested in this as it has been several months. I’m still learning too, but it seems that the Transformer architecture fits most closely into the transfer learning approach of ULMFit as a pretrained model has been made available. TransformerXL (as of now at least) does NOT have a pretrained model, so you have to go through a lot more work and GPU time to get to a nice starting point.

These other posts have quite a few comments and suggestions on how to use the Transformer* architectures. From what I’ve experienced and what people have posted, they require different hyper parameters and behave quite differently (training time, how accuracy changes over time, etc)

Fastai Transformer and TransformerXL

Are there any example notebooks showing how to use the new fastai Transformer and TransformerXL architectures? I have been playing around with them, but haven’t had much luck training with the default config. Accuracy goes way down after I call learn.unfreeze(). Are these implementations intended for text classification task? Should they outperform the AWD-LSTM and QRNN implementation?

Training TransformerXL

The purpose of this topic is to share experiences with hyperparameter for training the Tranformer XL. Here is my experience: the architecture uses a lot of GPU which makes experimentation more lengthy. To minimize restarting the jupyter kernel i have started to use stas’ “with gpu_mem_restore_ctx():” in the notebook when making grid search training time to “convergence” increase dramatically as the number of data increases training is very sensitive to hyperparameters setting and can diverge …