ULMFiT: Understanding learn.predict output


#1

I’ve written a test script for testing the classifier trained using train_clas.py (on a custom dataset).

Now I don’t have any OOB test data so I usually just send in the (only one) text string, tokenize it, load the classifier, run predict and evaluate the prediction(s) myself. Some like this:

tst_sent = tok2id(dir_path, tst_tok, max_vocab=30000, min_freq=1)
tst_ds = TextDataset(tst_sent, np.zeros(len(tst_sent)))
tst_dl = DataLoader(tst_ds, bs//2, transpose=True, num_workers=1, pad_idx=1, sampler=None)
md = ModelData(dir_path, None, None, tst_dl)
learn = RNN_Learner(md, TextModel(m), opt_fn=opt_fn)

prob = learn.predict(is_test=True)
# output: [[-11.48408  13.15376]] <-- what are these numbers?

pred = np.argmax(prob, axis=1)
# output: [1]

My question is simple: what does learn.predicts() return in the ULMFiT based classifier?


(Nick) #2

These numbers are per-class scores. The higher score is the result the model chooses.

If the classes are class_a, class_b you can get the result with something like this:

classes = ["class_a", "class_b"]
return classes[np.argmax(raw_scores)]

#3

Thanks for your reply. What I had meant to ask was: what function was used to get these scores?


(Nick) #4

It’s a PoolingLinearClassifier. See https://github.com/fastai/fastai/blob/master/fastai/lm_rnn.py#L175 for the code.


#5

Ah, right. So I just softmax it and I’d have a nice distribution in [0,1).


(Gerardo Garcia) #6

Imagine that already have the model and I just need to test a single prediction.
What is the best way to do this?


#7
  1. Tokenize your sentence (such as with the Tokenizer module)
  2. Map the tokens to indexes. The logic will be the same as here.
    … and the rest of it is the same as the code snippet in the question.

(Nick) #8

Use the (newish) script in designed for doing exactly this: https://github.com/fastai/fastai/blob/master/courses/dl2/imdb_scripts/predict_with_classifier.py