What changes to do when I shift to multi-GPU setup?

I have started using Google credits to work on GCP. I currently use a single GPU but would be using multi GPU after some time.

So I wanted to know what things to change in my code. Are there are best practices when using multi-GPU that I should be aware of?


  • Google Cloud Platform
  • PyTorch + fastai

Try this out.


I’d check out horovod: https://github.com/horovod/horovod#usage

Not sure how it compares to pytorch’s distributed package, but afaik it compares quite favorably to TF’s (https://eng.uber.com/horovod/)

1 Like

If you’re working with fastai, this (distributed) is the first answer.