Anyone have any success deploying ULMFiT on Sagemaker and using the model.get_preds() function?
Hi @efinkel, yes I have deployed ULMFiT on Sagemaker. Using learn.predict()
is what worked for me to get the predictions.
Hi all,
I am deploying a ULMFiT model using fastai v1.0.59 using gunicorn to host a REST endpoint. Can anyone suggest how to decide what values to set for the number of workers
for gunicorn
and number of threads for pytorch (torch.set_num_threads()
)?
I have noticed that the model latency is higher if I increase the number of threads and workers, but I am unable to figure out the optimum values to set for both.
Looking for advice around the same.
Thanks @alohia. I did actually figure out how to use .get_preds() which did improve my inference speed considerably. In case anyone is interested you need to set the following after adding a test set to your learner:
learner.data.test_dl.num_workers = 0
Can you share more on how you managed to use get_preds() in production?