One thing you can do is replace PIL with PIL SIMD which should speed up the Resize functionality. Also, slowly remove the fastai code and replace it with raw PyTorch (IE use fastai to build the DataLoader, but then use raw PyTorch to feed it to a model and convert it to the output you want), you should be able to speed it up via this method
While this is in fastai2, I have an example of what I’m generally talking about here: Speeding Up fastai2 Inference - And A Few Things Learned