ULMFiT: Understanding learn.predict output


I’ve written a test script for testing the classifier trained using train_clas.py (on a custom dataset).

Now I don’t have any OOB test data so I usually just send in the (only one) text string, tokenize it, load the classifier, run predict and evaluate the prediction(s) myself. Some like this:

tst_sent = tok2id(dir_path, tst_tok, max_vocab=30000, min_freq=1)
tst_ds = TextDataset(tst_sent, np.zeros(len(tst_sent)))
tst_dl = DataLoader(tst_ds, bs//2, transpose=True, num_workers=1, pad_idx=1, sampler=None)
md = ModelData(dir_path, None, None, tst_dl)
learn = RNN_Learner(md, TextModel(m), opt_fn=opt_fn)

prob = learn.predict(is_test=True)
# output: [[-11.48408  13.15376]] <-- what are these numbers?

pred = np.argmax(prob, axis=1)
# output: [1]

My question is simple: what does learn.predicts() return in the ULMFiT based classifier?

(Nick) #2

These numbers are per-class scores. The higher score is the result the model chooses.

If the classes are class_a, class_b you can get the result with something like this:

classes = ["class_a", "class_b"]
return classes[np.argmax(raw_scores)]


Thanks for your reply. What I had meant to ask was: what function was used to get these scores?

(Nick) #4

It’s a PoolingLinearClassifier. See https://github.com/fastai/fastai/blob/master/fastai/lm_rnn.py#L175 for the code.


Ah, right. So I just softmax it and I’d have a nice distribution in [0,1).

(Gerardo Garcia) #6

Imagine that already have the model and I just need to test a single prediction.
What is the best way to do this?

  1. Tokenize your sentence (such as with the Tokenizer module)
  2. Map the tokens to indexes. The logic will be the same as here.
    … and the rest of it is the same as the code snippet in the question.

(Nick) #8

Use the (newish) script in designed for doing exactly this: https://github.com/fastai/fastai/blob/master/courses/dl2/imdb_scripts/predict_with_classifier.py