ULMFiT in production

Deep Learning
1

Anyone have any success deploying ULMFiT on Sagemaker and using the model.get_preds() function?

1 Like
2

Hi @efinkel, yes I have deployed ULMFiT on Sagemaker. Using learn.predict() is what worked for me to get the predictions.

3

Hi all,

I am deploying a ULMFiT model using fastai v1.0.59 using gunicorn to host a REST endpoint. Can anyone suggest how to decide what values to set for the number of workers for gunicorn and number of threads for pytorch (torch.set_num_threads())?

I have noticed that the model latency is higher if I increase the number of threads and workers, but I am unable to figure out the optimum values to set for both.
Looking for advice around the same.

4

Thanks @alohia. I did actually figure out how to use .get_preds() which did improve my inference speed considerably. In case anyone is interested you need to set the following after adding a test set to your learner:

learner.data.test_dl.num_workers = 0

5

Can you share more on how you managed to use get_preds() in production?