How to make predict() run faster?

I am currently using predict() for image segmentation and it’s working quite well accuracy-wise. However, it is quite slow, taking around 80-90 seconds when provided with a 3000x3000 pixel image. An obvious solution would be to lower the resolution of the images, but that results in an unacceptable loss of accuracy. Is there any way to make it faster without sacrificing quality? Perhaps by somehow making it run on the GPU? Also, what are the major differences between predict() and get_preds()?

1 Like

A good question, I’m also looking into inference API at the moment, and didn’t find an easy way to run predictions in batch mode on the testing/new data. If I remember correctly, get_preds works on the attached data loaders, while predict allows running a model on the new data.

Also, 3000 x 3000 is a quite huge image. (Compared to “standard” 224x224 or some other similar sizes.) And, if you give a model an image of size that is different from the images it was trained on, you’ll probably get bad results anyway (or failure, depending on the architecture.) I think that fastai learners keep transformations and do the resizing under the hood anyway.

If you’re trying to predict a segmentation mask, you can go with a sliding window approach and make predictions on small patches, and then merge them together into a single mask.