Hello Jeremy and fast.ai community!
I’ve been working a bit with ULMFiT and I’m curious what is the preferred way to use the resulting LM or classifier model for inference. From what I’ve gathered, the preprocessing pipeline (tokenizer, vocabulary selection, numericalization) is a part of the DataBunch object, but I’m not yet sure how to extract all this into some kind of inference function which @devforfu mentioned.
I guess my ultimate goal/wish would be to perform something like:
data = ['sentence 1', 'sentence 2', 'yet another sentence', ...]
predictions = inference(data)
which probably is a common use case.
Any suggestions which components are required to build a function like this?
Also, another thing - I’ve tried to move the model to cpu by doing:
data_lm = TextLMDataBunch.from_csv(path)
learn = RNNLearner.language_model(...)
learn.fit(...)
X, y = next(iter(data_lm.valid_dl))
cpu_model = learn.model.cpu()
preds = cpu_model(X.cpu())
but the last line failed with RuntimeError: Expected object of backend CPU but got backend CUDA for argument #4 'mat1' - do you have any suggestions what steps are needed to amend this?
Thank you!