Just to share some personal experience: I recently bought a used Dell workstation and built my machine for < $350, and I've been rather pleased with the performance, considering how little it cost.
$125 - Dell Precision T3500 workstation (Xeon W3565 processor and 24G of RAM) (Craigslist)
$75 - Corsair CX750M PSU (new)
$140 - GTX 970. (actually, I already had a 970 sitting around, but this is what it'd cost on Craigslist/ebay.)
I installed Ubuntu 16.04 and used the setup script provided with the course materials. Although this PC can accommodate two GPUs, I did find it helpful to remove the original card (an nVidia Quadro FX 580), as its presence seemed to cause some trouble installing nVidia drivers / CUDA etc. I run this machine headless, so the GPU is 100% available for DL.
Using the Lesson 1 Cats/Dogs benchmark, this ran about 400 seconds per epoch on the full set. So far, I've not found the performance all that limiting. When I was doing the State Farm homework, e.g., I was able to experiment aplenty and got a result that was well into the top 1/3.
I do plan to replace the 970 with a 1070 soon; it will be interesting to see how much of a gain I get. It's certainly possible that the PC could bottleneck a more powerful card.
A word of caution: if you do go this route, you do have to replace the stock power supply.