The Vision D is a great board, but as it has been highlighted by the user above, it got 3-slot spacing between the cards. That mean you got to use a bridge for the A6000 if you want to NVlink them. Check (or ask Nvidia) for compatibility.
Take at least one pcie 4.0 ssd. Not that it will make much difference for DL-related tasks, but they are only marginally more expensive than 3.0, and they’ll make you system more reactive.
Prices are overinflated right now. Consider buying second-hand parts or previous generation parts, particularly the cpu (a r3000 will do just as fine as a r5000).
I figure the 3-slot spacing would be quite beneficial for thermals though? I would be quite surprised if the the 3-slot spacing NVLink bridge didn’t work for the RTX 3090. I’ll look into it.
Yup, good call. I’ll make that change.
Yup good call. I’ll consider swapping our for an R3000.
Thanks @balnazzar. Actually the 3090 is running at 14 TFops. However, the “older gens” were running at lower clock speeds. I’ve just upgraded my drivers and now all cards are actually running at the same clock speeds.
Phew, no need to replug anything (Andrea knows how much that scares me )
I’ve decided to water cool my 3090 Turbo as it is too loud. I’ve already received water block, pump, radiator, etc… but I need to get a new case as my current one is too small.
Do you have any recommendations for good cases for water cooling 2x3090? I have one 360mm radiator that is 45mm thick, and will add a second one (or a 240mm?) to the custom loop when I get my second 3090 later this year.
Thank you for all of the back and forth on this one. Just so that it’s all in one place, here’s my refined build list based on feedback and some other considerations. All prices in AUD.
Is there any comparison out there between 2x3090 with and without NVLinks bridge? I’m wondering how much will be the impact on deep learning applications.
I want the bridge so that I can train networks where even one dataset being trained at a time might need > 24 GB of RAM. So for me personally it’s a matter of the bridge increasing flexibility on what I can do, not so much a speed question.
You can do it even without the NVLink bridge. The difference it that the GPUs won’t be forced to communicate through the PCIe bus (that is, passing through the cpu, which slows down the process).
@dvachalek if you really put the gpu @350W under continuous stress (not timespy/furmark, and not even DL training, but gpu_burn or a mining sw) then the vrams will approach 110C.
Putting heatsinks upon the backplate does help, given that you have sufficient ventilation and case airflow, but not by much. The solution? Use your GPU at 260-280W. You won’t sacrifice much performance, and your temps will stay under 100C. Particularly if you already installed additional heatsinks.
Want to use the gpu at 350W no matter what? Then go liquid. Maybe install a ram waterblock on the backplate too… This will solve the ‘problem’ once and for all.
Remember that such a gpu is sold as a gaming gpu. And indeed Nvidia did cut 50W from the a6000, also using the slower gddr6. Note that the a6000 is not driver capped, but it is often inferior to the 3090 in DL benchmarks.
What I’m saying here is that two 3090s do cost almost half a single a6000 while providing a lot more perf even if used at reduced wattage.
Maybe a stupid question, but did you come across a way to monitor GPU mem temperature in Linux? Both nvtop and nvidia-smi are only reporting the core temperature.
By the way, if any of you is interested, after water cooling, my 3090 Turbo temperatures fell from 72 degrees to 43 degrees with a power limit set to 280 watts and 47 degrees without power limiting. It is certainly not cheap, but IMHO worth the cost just for the noise reduction.
My loop is very simple, I only have reservoir -> pump -> water block -> radiator. I have a 360x45mm rad. I’m going to add a second radiator as soon as I add the second 3090 to the loop.
To be completely honest, I don’t think I would have the time to install Windows on another partition just to check the memory temperature anytime soon — it’s a crazy time at work + one big personal project taking lots of my spare time.
I need to figure out how to reduce the memory temperature though as the backplate is quite warm. I might just try to measure the backplate temperature to approximate the gpu memory temperature sometimes next week. I saw people adding heatsink + fans and some even going all the way to installing a memory water block on the backplate. Maybe just a side fan blowing air toward the gpu on the back direction is enough.
The main thing for me is to find a solution I can use for both cards as I’m planning to add a second 3090 this year.
)