I’m currently looking into building my own local GPU server. Just wanted to hear if anyone else is looking into doing the same and if not what is you’re reasoning for keep using the cloud instead of building a local server?
For my part the main reason to build a local setup is to have a setup that allows me to experiment faster and that also incentivizes me to maximise time spend using the GPU (instead of minimising time spend using a cloud GPU).
Currently I’m considering buying a Geforce RTX 3090 TI and build a server around that.
Would love to hear thoughts and possible collaborate with people also considering building their own GPU server. Since, it is a sizeable investment (for me at least) I’m trying to do some research before hitting the “buy” button :-).
You may want to check out this thread. It’s still pretty active and has a lot of good info about building your own machine. I’m happy to answer any questions you have. It might be best for you to post them on this thread as others might be more likely to chime in.
I agree with this. I think the biggest thing is getting started is definitely easier on the cloud than building your own machine and can be much cheaper if you’re unsure how much time you will actually be spending training models, but if you’re in this for the long hall and plan utilize a machine regularly then building your own machine will likely be significantly less expensive. It also provides an opportunity to learn how to set up and maintain everything yourself.
I did this about a month ago I initially started with Google Colab and then switched over the Colab Pro plan (USD 10 a month, I think?). My initial analysis was that it would be $10 a month for Colab, but a good machine would cost around $2,500 - 3,000 to build. And that would come to about 20+ years of Colab usage at the current rate.
The pros there were that Colab would continue to update hardware at no cost to me and if I wanted to upgrade my own machine, I’d have to pay more and I’d certainly have to upgrade every 3 - 4 years (at least) if I wanted to keep up with improvements in hardware.
But where this broke down (at least for me) was in using Colab on a regular basis. There was always the initial set up time for every notebook. Then there was waiting for datasets to download and what not. I reduced some of this time by downloading the datasets to Google Drive and setting up each notebook to connect to Drive and use the files from there and also writing all my generated images to drive directly. This helped a bit but still having to set up the notebook each time ran was grating.
Plus, Colab was getting more use at this point due to Stable Diffusion and they were tightening down on usage, even on Pro plans. That’s when I decided to build my own machine.
It actually cost only about $2,000 (with some research and hunting) for a machine with an RTX 3090 with 24GB of RAM. I didn’t buy a monitor since I connect to the machine via remote desktop from my MacBook. I probably could have built it for slightly cheaper if I was in a different geographic location and was not trying to get parts I could get immediately rather than waiting a few weeks. I know I certainly could have built a slightly better rig for the same price if I’d been willing to wait …
So I guess it all depends on what you are willing to put up with from Colab and what kind of money you’re looking to spend. But if you have questions do let me know and I’d be happy to answer.
Amazon SageMaker Studio Lab came pretty close to this. It had persistent notebooks, supposedly faster machines than Colab, and it was free. You have to apply for access and wait about a week but then you have some fairly generous free usage. Unfortunately, while I could play around with a normal instance, I could never get a GPU instance when I needed one. I tried for several days (possibly) and then gave up and have not tried since.
It does have a very nice Jupyter interface, a fairly good amount of disk space and lots of useful resources (and even free courses) for Deep Learning.
And thanks for your thoughts on cloud vs local server Fahim . Really appreciate it!.
I have gone through lots of the same experiences as you. Essentially, I always ended up using quite a bit of time on setup on colab or paperspace. With paperspace I could have a virtual machine that I ssh’ed into, which automatically connected to a SSD with all my data but that quickly became somewhat expensive and then I had trobules of not being able to get GPUs, when I needed them.
I’ll try and create the same setup as you Fahim. Get an RTX 3090 with 24 GB RAM and then connect to the server from my MacBook.
If there is something you wished you had known before building the server with respect to components etc. then I would of course love to know .
Good luck with the build, I know it can sometimes be a bit unnerving till you get done — at least for me it was, because I kept second-guessing myself. Plus, I had the parts selected but at the last moment realized that I had to wait two weeks for an Intel CPU and so decided to switch to AMD and that meant that I had to switch a few other components out to match the new CPU
This site helped a lot since I could build a config on there and see if there were any compatiblity issues between components. So, if you didn’t know about it, it might be helpful:
In my case, in my hurry to switch over motherboards, I missed the fact that the new motherboard did not have WiFi built-in and so had to get a separate WiFi card since my machine is located far enough from the router not to be able to use wired ethernet. Might not be an issue in your case but that was one of the few snafus I ran into.
The only other useful but of info that might be helpful (or not) is that I used my TV as the monitor for initial set up. I bought a cheap mouse and keyboard combo to do the setup but after everything was set up, I hardly ever use the mouse and keyboard. I just turn the machine on and use remote desktop to connect.
If you have any questions about the process or run into any issues, feel free to contact me Would be happy to help.
This is my personal opinion but for me Jarvislabs has been the best value for money (and time). The experience was more frictionless than paperspace gradient and I could get the GPUs that I wanted.
I’m thinking about building a new machine but the energy costs for a 4090 setup make me pause. For me, even going to a 3090 would be a huge improvement over my 1070ti ancient setup. So I think I’ll probably go with 3090 (3090TI for me doesn’t bring a whole lot more to the table from an energy/price perspective). The money I save going 3090 instead of 3090TI, I’d put in getting a board with more lanes and with more RAM to push that through to the GPU)
Yeah! Sometimes having a local setup is pretty cool especially if you have a good internet connection, which is not what I have.
If you are worried about the energy cost, I don’t think you don’t need to worry much. The bump from 3090 Ti to 4090 is not that big. I assume you don’t use these 24*7
I initially started with Google Colab and then switched over the Colab Pro plan (USD 10 a month, I think?). My initial analysis was that it would be $10 a month for Colab, but a good machine would cost around $2,500 - 3,000 to build. And that would come to about 20+ years of Colab usage at the current rate.
(Though I guess it depends on your location.)
(thanks for the tip matdmiller)