How to create custom classifier head on top of LM?

fastai
1

text_classifier_learner is used for ULMFiT model, it use mean and max pooling output of LSTM.
if i donot use official text_classifier_learner, how to define custom classifier :
such as:

  • only use last output of lstm

  • add self-attention layer on LSTM for classifier

  • use CNN build on LSTM output

  • define own custom classifier loss function

  • …
    or more generally , how to change text_classifier_learner to any pytorch model ?

is there any implemented code or tutorial or advices ?

thans very much

1 Like
2

There is implemented code in the fastai library: you just have to define your custom head for the model like our LinearPoolingClassifier, then define your own function to get the model like this one. Then, you may have to create your own split for the model if the default text classifier one doesn’t work, but it should be all you need before adapting the code of text_classifier_learner.

1 Like
3

thanks a lot .
but I could not find the definition or pass of loss_function in these functions?
may i ask that where is the loss function definition part ?

4

The loss function is a flattened version of cross entropy (as learn.loss_func will tell you). There is just the RNNTrainer callback that gets in between, to return only the output (since the model spits more things).

2 Likes