I’m working through the deep learning course material and setting up a small cloud-based training pipeline for a personal project. I’ve been reading about the recent Microsoft outage reports and the follow-up saying no malicious activity was found, which left me thinking about how much slack to build into my own setup. My plan is to rent a GPU instance and maybe pull some datasets from cloud storage, but I’m not sure whether I should assume occasional multi-hour gaps or just short blips. I don’t have much experience with production reliability, and I’d rather not waste a week reworking my scripts later. For those who run training jobs on cloud services, do you build in fallback storage or a local copy of datasets, or is that overkill for a small learner project? Any practical habits you’ve picked up would help.
Update - that Microsoft outage recovering, no malicious activity found piece made me think about keeping a local dataset copy, though I’m still unsure it’s worth the hassle for a short course project. Might just snapshot my training data before each run and see how it goes.