Cloud GPU services vs having your own, which is better for AI?

[matdmiller (Mat M)]

My experience is that running on your own machine can be significantly faster than an equivalently spec’d card in the cloud, but it has been several years since I’ve run things in the cloud. I think the 3090 w/24GB of VRAM provides good bang for your buck, especially now that prices have dropped. It all depends on how much you’re going to use it, but I made the switch after several $200/mo AWS bills. Going from a K80 to a 1080ti was dramatically faster, I think somewhere around 3-10x. If you’re curious I can run a speed test on a dataset on my machine so you can compare it to running in the cloud. I did run a test on the IMDB language models and posted the results here:

Colab model learning speed

A larger batch size is usually faster than a smaller one because the GPU utilization is higher, although that’s not always the case and depends on where the speed bottleneck is in your training pipeline. As a general rule of thumb you want your batch size to be as large as possible while still fitting on the GPU ram and divisible by 8 (if possible if you’re utilizing tensor cores). How long was your old vs new epoch time? I believe Colab will also sometimes give you a different GPU depending …

The results will vary based on the model type. I didn’t note which GPU I got in COLAB when I did this test, but I’m guessing it was a T4 or P100. It was not a K80.