Making your own server

Get at least as much as your total GPU RAM.

I put 64 in mine and that’s been more than enough for 2x1070, probably would have been fine with 32.

If you get a fancy CPU with many cores I might go higher on RAM to store all your preprocessed data.

@davecg Please explain why as much ram as GPU, aren’t they memory self sufficient or is it that memory is required for parameters etc. I guess it makes sense as something has to hold the data to feed and read the GPUs.
I am looking at refurbished windows7/10 workstations with high specs but without GPU and Disks. I guess I’ll have to bite the bullet and buy one.

Yeah, you need to be able to hold the data you’re feeding to your GPU in system RAM to avoid a bottleneck. That said a lot of the GPU RAM is taken up by intermediate layers and gradients which may never need to be in system RAM, so maybe you don’t need even that much.

Personally, I just put a full 64 Gb in my system so I can fit smaller datasets completely into memory (which has a decent performance advantage).

Has anyone considered this option (just $600)?

You add the monitor, keyboard, mouse of your choice, and use a USB thumb drive or SD card for additional drive space.

1 Like

Another option is to find someone selling their gaming rig on Craigslist. I did this and was able to pick up a pre-assembled computer with all the specs I wanted (including a 1080) for a decent price! Still requires a bit of setup (installing ubuntu, setting up deep learning libraries, etc) but it does save some time and money!

3 Likes

I just use the CUDA_VISIBLE_DEVICES environment variable for this purpose, FYI.

2 Likes

@brendan and @sravya8 Thanks for the awesome posts.

I am now inspired to put my deep learning box together. It would be great if you can clarify a couple of things for my understanding?

  1. Is the RAM expandable in the future if needed?
  2. I see that you have one GTX 1080. I was wondering if the rest of the hardware you have enables adding a second GPU (say GTX 1080 Ti) down the line?
  3. More generally how many GPUs can the setup you have accomodate?

Thanks

If you purchase 16GB chips, then yes. If you get 8GB chips, it highly depends on your motherboard. But most people will never need more than 32GB of ram.

You should be able to use two 1080Ti with most motherboards.

Some motherboards can do 4 GPUs, The last one will generally run slower and depending on the motherboard 3rd one maybe slower as well. As long as you are at 8x+ you are good, when it drops down to 4x (8x,8x,8x, 4x) you will have a more significant drop in performance. I don’t know how this changes with the 1080Ti, for all previous boards the difference between 16x and 8x was less than 1%, there might be more of a difference dropping to 8x with the 1080Ti.

1 Like

Thanks for your reply Christopher. Just so I understand: The parts list provided by @brendan has capacity to have 2 GPUs on it?

Another question is do the two GPUs have to be the same model? I am planning to get GTX 1080 now and the GTX 1080 Ti later.

His board can do 3x GPU, but one will run at 4x. I am not sure the performance hit of 4x, as I have only tested with as low as 8x.

For SLI gaming, you can’t mix a 1080 and 1080Ti, but I believe you can with machine learning. I never tried it so I can’t speak for that but I am 99% sure you can.

Thanks again Christopher.
I am a noob and so pardon my question: “3x GPU” in your reply means I can have 3 GPUs if I wanted to. What does “one will run 4x mean”? Apologies for pestering you with this.

3x GPU means you can have up to three GPUs in the system.

As for 4x, this is the speed of your GPU on the PCI Express bus. This means it is going to get 4 lanes (4x) of the PCI Express bus. For PCI Express 2.0 this means you will get 2GBps, for PCI Express 3.0 (rather new) you will get 4GBps.

For a single and dual GPU system, you want to be using 16x slots. Once you get to 3 and 4 GPU this isn’t possible anymore in most systems. So you start running something like 8x, 8x, 4x. Where the first two cards are running at 8x, and the last card only gets 4 lanes. The 8x isn’t a problem, in testing the performance difference between 16x and 8x is almost non-existent (sub 1%). Dropping down to 4x though may pose problems, especially with a 1080Ti.

I have no tested the 1080Ti or 4x performance, with PCI Express 3.0 it may even less of a problem as the bus speed doubled. I’m sure someone can do some testing when they have more 1080Ti’s in the wild to what it means in raw performance.

If you think you are going to run 3 or 4 1080Ti’s, I would highly suggest going with a Xeon build. This will allow a motherboard with 40 lanes.

A Z270 will allow 20 lanes, the older Z170 will only allow 16. The advantage of the Z270 is the ability to run two GPUs at 8x while still having 4x for an NVMe drive. Keep in mind, each NVMe drive will use up 4 PCI Express lanes.

4 Likes

Do you think it would be worth it to get a 1080 Ti as a third card at 8x if I would then need to run my two 1070s at 4x? (Intel 7700k).

Otherwise anyone in the market for a lightly used 1070? :slight_smile:

Also curious if you have any advice on Xeon processors and motherboards for a future build.

EDIT: Found the answer…

Looks like a 4% drop in performance at 3.0 4x vs 3.0 16x with almost no difference at 3.0 8x for a 1080. No benchmarks for 1080 Ti yet. Guess I’m getting another card. :smiley:

2 Likes

More RAM is very helpful for training larger models, since you can read your dataset into memory! :slight_smile:

2 Likes

Thanks a bunch Christopher. Your informative reply made me understand more of the alphabet soups I see on the specs pages. Again thanks for your patience in explaining this to me.

We talking GPU memory or system memory? I was referring to system ram, and unless your running 4 Titan X, 32GB with an easy upgrade to 64GB (two slots still open if you stick with 16GB chips) should be enough right?

Think he’s talking system memory.

Point was if you have enough memory to fit your dataset completely in RAM, you will see a significant speed up (RAM > SSD > spinning drive).

So doesn’t really matter for ImageNet, but for many kaggle competitions it will speed things up if you have a full 64 GB.

I went with Intel Xeon E5-2620 V4 2.1GHz 8-Core Processor because it can support 40 PCIe lanes and a lot of memory. I went with a MSI X99A TOMAHAWK ATX LGA2011-3 motherboard because it also supported 40 PCIe lanes, has space for up to 3 GPUs, and up to 128 GB of memory (8 slots).

I started with 32 GB of ram, but might expand that in the future.

I’ve only recently set it up and haven’t done much beyond cats and dogs, but so far it is working well. I’m planning to start state farm distracted driver soon.

1 Like

You think correct :slight_smile:

Even for ImageNet it can be helpful, for saving pre-computed activations from the last layer (fits in 128GB RAM easily).

Using a ram disk?