UMLFit text_classifier on sentence pairs

manish · July 1, 2019, 11:41pm

Has anyone tried ULMFit text_classification using text input pairs?
Instead of single text classification by sequencing output of encoder and PoolingLinearClassifier we would have separate encoders for each text thats part of input and concatenate the output of encoders before passing it to PoolingLinearClassifier.
Thoughts?
We could hack it by concatenating the two texts as single stream and fitting a simple text_classifier_learner but it would be harder for model to know the separation of the texts in this mode.

sshleifer · July 4, 2019, 2:11am

Haven’t tried it but this seems relevant:

github.com

arvind-cp/fastai/blob/arvind-cp-LM-eval/courses/dl2/FiTLAM - Quora Duplicate Pairs - STS task.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FiTLAM Eval for Quora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/envs/fastai/lib/python3.6/site-packages/matplotlib/__init__.py:1067: UserWarning: Duplicate key in file \"/home/ubuntu/.config/matplotlib/matplotlibrc\", line #2\n",
      "  (fname, cnt))\n",

This file has been truncated. show original

see LinearComparator

Thread discussing results: What does the encoder actually learn? 🤔

johnmarkl · March 10, 2020, 12:47pm

Bumping this.
Any simple way to build a text classifier having two separate texts as input?

morgan · March 10, 2020, 4:09pm

I’d recommend looking at some of the winning solutions from Kaggle’s recent QUEST Q&A competition: https://www.kaggle.com/c/google-quest-challenge/discussion/129844

Some of the winners had 2 BERT models which combined the last BERT layer of each into another linear layer or two. One BERT model was trained on questions, the other trained on answers. Some have shared their code too if you need inspiration.