What changes to do when I shift to multi-GPU setup?

Deep Learning
1

I have started using Google credits to work on GCP. I currently use a single GPU but would be using multi GPU after some time.

So I wanted to know what things to change in my code. Are there are best practices when using multi-GPU that I should be aware of?

Setup:-

  • Google Cloud Platform
  • PyTorch + fastai
2

Try this out.
https://docs.fast.ai/distributed.html

2 Likes
3

I’d check out horovod: https://github.com/horovod/horovod#usage

Not sure how it compares to pytorch’s distributed package, but afaik it compares quite favorably to TF’s (https://eng.uber.com/horovod/)

1 Like
4

If you’re working with fastai, this (distributed) is the first answer.