I trained an ULMFIT model on Google Colab GPU. When I test predictions on Colab with GPU, I get about 1000 short text documents/second. On Google Colab with CPU, I can predict at about 140 documents/seconds. However, on Windows 10 with CPU with the same model and same code, I can predict only 18 documents/second, so Colab CPU is 8x faster.
On Windows, I get this ominous warning
C:\Temp\hug\lib\site-packages\fastai\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input warn('Tensor is int32: upgrading to int64; for better performance use int64 input')
On both Colab and Windows, I use torch version 1.4.0 and fastai 1.0.60.
Any advice on how to speed up Windows to be more like Colab? Running on Windows would make deploying my model easier because my IT department prefers Microsoft.