GTX 2080/2080Ti RTX for Deep Learning?

[antorsae (Andres Torrubia)]

Just compiled pytorch w/ cuda 10. Logs showed compilation of 7.0 (volta) compute capabilities and not 7.5 (turing). Not sure if this will bite me later or not…

log of python -c 'from fastai import *;show_info(0)' is:

distro info    : Ubuntu 16.04 Xenial Xerus
python version : 3.6.3
fastai version : 1.0.5
torch version  : 1.0.0a0+805f4d5
nvidia driver  : 410.57
cuda version   : 10.0.130
cudnn version  : 7301
cudnn available: True
torch gpu count: 2
  [gpu0]
  name         : GeForce RTX 2080 Ti
  total memory : 10989MB
  [gpu1]
  name         : GeForce RTX 2080 Ti
  total memory : 10981MB

Is there any standarized benchmark for fastai?

[init_27 (Sanyam Bhutani)]

Maybe try asking in the #fastai-dev section?

[jeremy (Jeremy Howard)]

Unfortunately no. Check out:
one of the best speedups in benchmarks I’ve seen is 50% when switching to fp16 from fp32

We have the current DAWNBench record. We’re seeing it about twice as fast. Try it with fastai’s fp16 support.

[devforfu (Ilia)]

Does it work for all kinds of tasks? I mean, it should give a speedup for various kinds of models, and not only to image recognition architectures, right?

[miwojc]

adding another card to double it’s memory

Is that the case? I’ve read somewhere that it will not double the memory if you have two cards even for 2080ti

[devforfu (Ilia)]

Well, it should if you train two separate models in parallel, I guess =)

[jeremy (Jeremy Howard)]

I mean, it should give a speedup for various kinds of models, and not only to image recognition architectures, right?

It won’t help as much for RNNs, although possibly QRNNs it will - I haven’t tested. I also haven’t tried localization tasks.

[jeremy (Jeremy Howard)]

I’ve read somewhere that it will not double the memory if you have two cards even for 2080ti

If you use nn.DataParallel then effectively you get double the memory. You should use NVLink to get good performance, of course (which requires a 2080 or better IIRC).

[devforfu (Ilia)]

Hm, understood. Thank you for the response!

I guess it is worth to think more carefully about what GPU to purchase than =)

[Ralph]

NVLink is available on 2080 as well, but not 2070.

[liberus (Andrii Boichuk)]

Do you have some kind of comparison table? Or some kind of benchmark to test?
I knew that you had nice record there, but I missed details of the implementation, if you somehow make fp16 twice as fast as fp32 than it really changes my video-cards comparison

[devforfu (Ilia)]

I guess the source code used to compete in that benchmark is available via this repo.

[Even (Even Oldridge)]

So, finally, if you were choosing between 1080Ti and 2080 (plain), which one would you prefer? Or, if take into account the whole range of cards: 1080Ti, 2070, 2080, and 2080Ti, which one is the best choice considering their computational power, memory, etc?

1080Ti would be my recommendation here. Memory is one of the more important aspects to deep learning, and while there are ways around the limitation now it’s still complex. 11gigs of memory is a much better footprint over the 8gigs of the 2080, especially when you’re also running the graphics off that card as well.

regards
Even

[jeremy (Jeremy Howard)]

This is the updated imagenet training repo https://github.com/diux-dev/imagenet18

[balnazzar (Andrea de Luca)]

1080Ti would be my recommendation here. Memory is one of the more important aspects to deep learning, and while there are ways around the limitation now it’s still complex. 11gigs of memory is a much better footprint over the 8gigs of the 2080, especially when you’re also running the graphics off that card as well.

I still got to understand how much memory a rtx card spares when it runs computations in mixed precision, both for vision and for structured data. That is, how much of that mixed precision computation is actually done in FP16.

[Ralph]

I’m seeing 2070 as low as $499 at EVGA so you could get 2 8gb fp16 gpus cheaper than the single 11gb 2080ti. The 2070s do not have NVlink, but I don’t know if that would add enough benefit to offset the increase in power/memory at a lower price.

[devforfu (Ilia)]

Hm, I see. Yes, I am thinking right now what is more beneficial: GPU with fp16 computations support, or 11GB of memory. Haven’t thought that the answer would be so difficult

[matdmiller (Mat M)]

I’m interested in this answer as well. @sgugger has an excellent post on single precision vs mixed precision. I just posted a question on the Mixed Precision thread asking him what memory usage he’s seen in practice on the mixed precision work he’s done.

Mixed precision training

Thanks for the awesome explanation. What does your overall memory consumption look like when using mixed precision vs single precision? Are you able to increase your batch sizes in practice or does keeping a single precision copy of your weights cancel out the single precision gains w/ respect to overall memory usage. I’m sure it varies based on which model you use, but any insights on what you’ve actually seen in practice would been interesting to know.

[PeterKelly (Peter Kelly (OZ))]

Hi Ilia (hope name is ok). I bought one last week in preparation for Jeremy’s new course tomorrow. I HATE games (grumpy old man). I will be looking to advise you numerically, as the machine that I am ‘uograding’ is old i7-9?? in lga1366, x58 board. I’ll be incrementally plugging in Gigabyte 2080Ti oc, 970evo vnand ssd, and 2Tb 860 vnand pro: should be fun and instructive. Will advise.
Cheers, peter kelly.

[PeterKelly (Peter Kelly (OZ))]

I’d have missed that !! Thanks Jeremy.
P