I was wondering about the best way to include additional features to text_classifier_learner. There are two cases
Numerical Features
I feel like the most natural solution would be to concatenate the output of the language model encoder with new features. To do that we would need to create a custom classifier head. I found this post discussing it
Text Features
Adding features like the title or name of the category. For example, film genre when we want to predict it’s IMDB score based on the review. The easiest way to handle that would be to combine all the text features into one and add the new feature words into vocab. The problem I have with this approach is twofold: we can lose some information because the name of a genre and word in a review might be interpreted differently, and secondly, we make the problem we are solving more complex in comparison to having genre information given explicitly.
Distinguishing these text features among other words seems like a reasonable approach to address these problems. Maybe converting features into special tokens and initializing their weights to original words would be a good idea. Then the meaning of the category could be updated separately from the meaning of a word.
“Horror”
“It was more of a comedy I think than it was a serious movie”
into
“_horror_ It was more of a comedy I think than it was a serious movie”
Any thoughts?