I thought I’d post here about my 2x 3090 rig, since i’ve seen very few builds with 2x 3090 documented.
Case: Lian-Li O11 XL
MB: ASUS X299 Sage
CPU: i9 10940
Memory
GPUS: 2x3090-FE
PSU: Corsair AX1600i
CPU cooler: LianLi AIO
I selected this motherboard so that i could get 4-slot spacing for the 3090s, allowing a larger air gap between them, per Tim Dettmer’s blog.
The case, i guess is a splurge but also was great to have the space for cabling and circulation.
I had started this build using a 1200W Corsair. This did NOT work for 2 GPU training workloads. As soon as i would start certain types of training workloads, the machine would immediately power cycle. I could prevent power cycling by locking gpu clocks (nvidia-smi -lgc 1600) to something below peak clocks. 1600mhz worked well for me, YMMV. I didn’t have success with wattage limiting using nvidia-smi and i think the reason is because 3090 has power spikes even if you wattage limit, but by limiting the peak frequency, the spikes are reduced. On many training pipelines, this would result in a pretty minimal reduction in overall speed, but i wanted to be able to run the 3090’s flat out, so i migrated to the 1600W PSU which has solved the power problems.
After resolving the power problems, the system’s been a dream.
At flat out, the lower GPU maintains around 63C and upper GPU around 75C. I will probably invest in one more set of fans to the right of the GPUs to provide more cool intake. The lower fan set is pull and the upper AIO is push, the rear fan is push.