I agree wrt having your own card vs cloud compute. So handy to have your own rig, and gpus hold their prices well so you probably won’t loose more than 1/3 to 1/2 the cost (depending how long you keep for).
Having 2 x cards (or 1) in machine is fairly straightforward, but if you plan to have more than 1 gpu and if you can, get a motherboard that has 3+ double spaced pcie slots so you can have some room between the cards. (Alternatively 2 x blower style (each 2xpcie slots wide) can fit right next together).
Gpus that are 2x pcie slots wide are more manageable than the ‘extreme’ editions that can be 2.5 pcie slots wide (unless you find these really cheap).
Having 3 or 4 gpus in one machine is a headache -blower style cooling may be fine but still need psu with sufficient power and decent cpu , air cooled can do 3 with big case and good fans, 4 probably getting too hot with air cooling unless you use pcie risers, custom water cooling is a time consuming project, AIO’s may be good option-but need to have enough room for the radiators. i would do 3 or 4 gpu setup as a second build once youve built a 1 or 2 gpu box. If you do want a 3/4 gpu rig I would select motherboard first, then plan build around it. If air cooling -ve pressure in the case has worked best for me.
Cpu wise 4 core fine for 1 gpu, another 2 or 4 cores would be handy if have second gpu. 32 gb ram is nice to have per gpu, but can get by with less.
GTX1070/80/80ti/RTX2060/70/80/80ti all decent choices depending on your budget and their price. I found online store here in au selling galax 2080ti model that was superceded at great price.
Re used gpus: I prefer to buy used gpu from source where can see running and pickup from owner rather than getting posted.
Cpu’s i have bought online via ebay (used)
Motherboard i prefer new (ebay) but have also bought used locally.
Ram i mostly get used from ebay
HD/ssd/nvme id only buy new (local store)
PSU i go with gold or better rating ideally new.
Case, fans, buying used can save a fair bit.
If you are after a used decent box that can fit 1 or 2 gpus the lenovo s30/d30/hp 420?/820/dell equivalent you may be able to find at a good price (though they all have motherboards with 2x pcie spacing and can only fit 2 cards max so they limit what gpus you can get)-or get a used gamer box without gpu.
I have only run gpus on own (each training own model) or using pytorch dataparallel, haven’t tried nvlink yet
I have had no trouble with any gpus i have had and think it reasonably unlikely your gpu will die unless you get one without seeing it running and checking temp under load/or without some knowledge of its history.
In order of items that hold their price i think: gpu>cpu/motherboard/ram>case/fans>drives/psu