Don't use Paperspace Gradient

Part 1 (2020)
1

This post is to warn the fast.ai community about the risks of losing data when using Paperspace Gradient.

This is the second time Paperspace Gradient has lost my data. Below are the details:

Recently, Paperspace upgraded its notebook instances from V1 to V2. Soon after, my V1 notebook instance refused to start. When I contacted customer support, they told me to update my notebook instance to V2.

When I followed their instructions & updated my notebook instance to V2, all my data in the persistent storage disappeared. After a few email exchanges, they finally replied this:

Hey Sambit, We’ve now had three senior engineers look – it’s honestly not a complex path at all. This particular data was not moved, and also happened to exist right around the moment our Cloud team did a purge of V1 data, which we have to do regularly to have ample space for everyone on V2. We cannot apologize appropriately or enough for this situation, and we fully understand your incredible frustration, disappointment, etc. It appears there is nothing we’ll be able to do to make it right, which is the worst part. In any case, I’m also sorry to be the bearer of this bad news. If there’s anything at all I can do, juts let me know. Best, Christopher Barosky Head of Customer Ops

That instance contained all my data from August 2020 (including class notes, edited notebooks, my own notebooks & fine-tuned models). This has caused me a tremendous amount of stress & heartache.

And this is the second time they have lost my data. (Last time, one notebook disappeared. This time, everything disappeared!)

So I would advise the fastai community to avoid Paperspace Gradient like the plague. And if you must use it, then please download your files to your local computer after each & every session. Otherwise, your files are likely to suddenly vanish into thin air!

1 Like
2

Accidents do happen, and I have traveled a similar path (Using my personal server and on some cloud providers). To avoid frustration, I started initializing the project in Github before I start a new one and make sure I commit every time before I take a break. So if I come across a system crash or lose data, the loss is always minimal.

2 Likes
3

You call it accident. I call it negligence. They should have backed up V1 instance data before converting V1 to V2.

If you search on the web, you’ll find that Paperspace has a habit of losing customer data. It’s not just me.

4

I was just mentioning it from a personal perspective. I was not justifying what happened.

5

It’s a good tip. Appreciate it.

But imagine having to git commit after every session of using Google Drive / Dropbox / OneDrive. :slight_smile: My mistake was assuming that Paperspace was as reliable as Google / Dropbox / Microsoft.

But anyway, I learned the hard way that “Your data will disappear.” should be the default assumption for all cloud providers.