Build your deep learning box: wiki thread


(Jeremy Howard (Admin)) #1

For those that have made it this far, we recommend building your own box, if you can afford to. For a bit over US$500 you can build something that should be faster than an AWS P2 or Google Colab server. For $1500-$2000 you can build something that’s about as good as a high-end Paperspace server.

This is a wiki thread. Please add resources that are reasonably up to date, and that you have personally read and found helpful. (If you see resources listed here that have incorrect advice, please add a note next to them mentioning those issues.)


So you want to build a Deep Learning AI computer system?
Building deep learning workstation for $2200
Cuda error 45: no kernel image - Pytorch 0.3.1.post2
About the Part 2 & Alumni category
(Gokul) #2

Thanks Jeremy for the resources. I’m building my own box too this week. Here is my list if anyone is interested
https://pcpartpicker.com/list/KX7gLJ


(Jeremy Howard (Admin)) #3

Thanks for the PC Part Picker link - could you tell us a bit about your thinking behind this build? What resources did you find helpful? How did you decide on this mix of RAM/CPU/GPU for this budget?


(Gokul) #4

Investing in GPU right now seems expensive (1080ti is upwards of ~1.2k). Expecting prices to drop down a bit when Nvidia introduces their new Ampere cards ( expected April https://www.rockpapershotgun.com/2018/02/26/nvidia-ampere-graphics-cards-release-date-rumour/).
Wanted to get a good processor, the difference between i6700k and i7700k(released q1-2017) was very minimal(http://cpu.userbenchmark.com/Compare/Intel-Core-i7-7700K-vs-Intel-Core-i7-6700K/3647vs3502). The improvement in performance between i7700k vs i8700k was better (seems like a worthy investment to make). Thumb rule for ram seemed to twice the vRAM
I’m hoping to run initial models with this box and then do the actual training over P6000 instance of paperspace (if things don’t work out locally)


(RobG) #5

I built a dedicated box earlier this year with a Titan XP I ordered from nvidia (uk) which was a good deal but a 1080 would have been just fine. The last one I built was in the late 90’s. Pc Part Finder was a great site to detect incompatibilities. I built it in an afternoon, but it took a couple of sessions to decide on parts. Part availability was a significant factor in choice.

The various blogs give good info, generally. What can I add beyond these blogs? It’s not difficult and you don’t need to be overly technical, it is essentially fiddly assembly. A bigger case is easier to work inside, if you have the room to put it, and may stay cooler. Decide how much you care about noise and understand the quietness of your case and any fans/coolers you buy. I ended up with BeQuiet parts. Think about buying a power supply with enough wattage and motherboard that can take another GPU later. You can add these on pc part picker to check future capacity. Consider keeping your OS on a separate disk to your data. Watch some videos on YouTube/Vimeo for your case/mobo if you can find them. I found watching the videos invaluable for fiddly bits - eg I reversed the side of my mobo and would never have worked that out on my own or with a mere pdf manual.

The most difficult part for me was having to find/buy a couple of adapters to hook up an old screen to do the initial config, and then working out on Ubuntu forums how to configure WiFi, and how to put my user directory on a 2nd disk. All plain sailing from there. Since this is a wiki I’ll see if I can find helpful links for those.


(William Horton) #6

Posted this in a different thread, but here’s my setup. Been really happy with it so far!


(adrian) #7

Current build:

CPU: i7 7820X (LGA2066)
CPU cooler: NZXT Kraken X52
Mboard: MSI X299 TOMAHAWK AC ATX LGA2066 (ASRock X299 Taichi gives 2x16 1x8 for 44 lane cpu, was a tough couce between these…)
RAM: Corsair - Vengeance LPX 32GB (2 x 16GB) C16 DDR4-2666 Memory
PSU: Corsair RM1000i
HD: WD Gold WD1005FBYZ 3.5” 1 TB
SSD: Samsung 1TB
GPU: Gtx 1080 ti
Case: Dark Base Pro 900

Comments:

I tried to put a gtx1070 (from another machine) in with the 1080ti. Right next to each other the 1070 was hitting 90 degrees and 1080ti hitting 85 degrees before I shut down and pulled out the 1070.

Plan now is to water cool the 2xGPU’s in one box, the main worry being that I get a water leak and destroy some very expensive equipment. (reservoir in 5.25" drive bay looks like best option to conserve space)

I found that there isn’t that much info on multi GPU deep learning builds - how to keep cool with just airflow, which are best motherboards, cases, how to watercool in ATI mini towers when you don’t want some fancy gaming box with lights everywhere and giant coloured reservoirs.

After inspiration from @tensoralex below I looked into cooling options with submersible pumps and found the following from AquaTuning a good value option:

NexXxoS ST30 420mm radiator
Alphacool Double Power Pumpstation
2x Eheim 1000 pumps (run in parallel with one at say 50% for redundancy).


(Alex) #8

Usually it wont fail if you make sure tube connectors are tied up. But better to place reservoir below below your GTX…just in case.

I figured custom made water cooling is much cheaper than similar grade systems (like 500 - 600$), and 100$ water cooling stock systems can barely keep up with GTX powers…


(Vikas Bahirwani) #9

I use my own box with GTX 970 (built it last year). I am wondering if it is sufficient (say for Part 2 Fast AI for now) or I should upgrade to GTX 1080 ti. (luckily my power supply is work if I do decide to upgrade :slight_smile: )


(Jeremy Howard (Admin)) #10

GTX 970 will work fine. It just depends how patient you are :slight_smile:


(William Horton) #11

Does anyone have good advice on how to set up two disks (one fast SSD, and a bigger, slower HDD) in Ubuntu? I put off the issue when I first built my machine, but I’m getting to the point where all the datasets for part 1 and part 2 don’t fit on just the SSD anymore.


(Vikas Bahirwani) #12

:slight_smile: I might test my patience for now and see when it breaks.

(Also I have never upgraded before so I hope that replacing the GPU will work without requiring a complete reimage of my Ubuntu followed by fresh fastai setup)


(Sanyam Bhutani) #13

With the next gen GPU release of Nvidia around- Do you think it is good to wait for a bit or should I instead dive in?


(adrian) #14

I presume they are both plugged in to your SATA slots - ie internal drives?

you’ll need to change /etc/fstab to be able to automount the new drive.

Just be carefull with fstab as pc wont full boot if you get it wrong and you’ll have to muck around to get working agin


(Vikas Bahirwani) #15

Based on the links that Jeremy gave above, I liked the recommendation of starting with GTX 960 / 970 and then selling it to get the latest model after.

You may want to create a box that could support a good GPU like 1080ti or titan and then buy something cheaper first.

OR you could get 1080ti now given that you are not a noob in deep learning space anymore :slight_smile:


(Sanyam Bhutani) #16

1080Ti is also going through a surged price everywhere (Because of miners). I think that Might be a good reason to wait.

I’ve been waiting patiently to sum up the money for getting my box since last November. (Part1v2)
Still a little far from it I think. :sweat_smile:


(adrian) #17

Usually is what I’m worried about :slight_smile:

What sort of watercooling setup are you using? ie what pump, reservoir, soft tubing? How do you fill a reservoir when its on bottom of machine if it needs a top up?


(Vikas Bahirwani) #18

I can understand. I think Paperspace or AWS could be fine for now at least for the class. For what it’s worth - I have 970


(kanishkd4@gmail.com) #19

@init_27 @vikasbahirwani
In india, look up assembled machines - much cheaper than the sum of all parts.
Also, the right credit card offer can save you about 15-20%


(Alex) #20

Just made a picture, in case you will be still interested I can provide all the components later. Around 100$ in materials.

Refilling becomes quite a maintenance once in 4 months, but other than that works great for close to 2 years…no leaks.