I’m on the 2nd lesson and I’m already pushing my computer and its GPU to its limits (returning a lot of memory errors at this point.) So I’m considering using Paperspace since they have several options for GPUs. But I’m still a little skeptical if the GPU+ instance that Jeremy recommended will cut it. First, its GPU is only 8 GB and my computer’s GPU is 4 GB and I was already getting out of memory errors by the 2nd lesson. Also, many of the fitting steps in Lesson 2 took forever locally, like almost an hour or so (I didn’t time it so I’m just estimating.) Is training instantaneous on the P5000 or do you still typically need to wait a few minutes?
Hello, I am using P6000 at the moment but it still needs a few minutes to train a model. For example, Dog Breed Classification has about 10,222 images and typically with resnext101_64 it takes about 7 mins for three epochs (with precompute=False). With GPU+, I had to reduce a batch size quite often (due to memory errors) but I did not have any issue since I changed to P6000.
I’m getting out of Memory errors on the NLP model in lesson 4 using the P4000. I tried to upgrade from Paperspace but it won’t let me, I’ve contacted support because it’s going to be pretty rubbish if I have to do a whole new machine and pay an extra month of their disk/IP charges.
I’ve tried to sign up for the 5000 but it seems it requires manual approval. I sent them a message that I’ll be using it for Fast.ai yesterday (Saturday) but it seems it hasn’t been read yet… maybe it’s just the weekend.
I sent them a message again through a Support button (at the bottom right side of the web page) after Paperspace didn’t get back to me for a couple of days
Hi guys, Dillon here from Paperspace. Really sorry for the delay we are getting to the approvals as quickly as possible. Thanks for being patient with us
I’m considering using P6000 too. Don’t wanna run into problems mid-course. Having taken course-1, AWS p2 instance cost 90 cents as well. P6000 should be safe for us, I guess.
Based on my experience, I would recommend P5000 to start off with.
I guess, probably move to P6000 once you want more speed and you are bit more advanced. I haven’t tried P6000 yet as I am in the early phase of the course.
FYI: Though they gave access to P5000, upgrade option from P4000 to P5000 didn’t show up. I had to deactivate and create new machine again. Doesn’t take much time to setup though.
I contacted support about upgrading the GPU not working and they said you can’t do it - it has to be a new machine. However the public IP/disk cost is prorated over the month down to the ms just like compute, so you don’t pay extra by switching (I was a bit annoyed at the prospect of extra fixed costs).
Also when you ask for approval, I’d suggest explicitly asking for the P5000 and P6000. I initially asked for only one expecting it to be a boolean approve/not, but it seems to be per instance type.
I emailed their support again and they said that they’re kinda behind in approving new P5000 instances right now… it’s been about a week and no P5000 so I guess I’m gonna go with Crestle for the moment.
Yes. Every penny. It’s 10 times faster than p2 Xlarge. The cloud desktop is great. You can monitor your run’s progress from mobile. I think you can do the same from crestle. But not in p2/p3.
@Deb, Could you please elaborate more on how to monitor your run progress on Mobile. Currently, every time I run an instance on Paperspace I have to leave my web-browser and computer on. Anytime, the network is disconnected I lose my progress and I have to start from the very beginning. In short, what I am looking for is how to set up the notebook or script to run in the background (when I know it is going to take a while to finish the training). Thank you so much for your help!
In paperspace I have turned off “no-browser” option in jupyter notebook config file. So that I can launch in both in the cloud desktop or through an ssh tunnel with --no-browser option from command line. To use it in mobile you need to use it in the browser inside the cloud desktop (not through the tunnel). This also allows you to launch jobs inside a (company) proxy wall. Also ensure the auto turnoff desktop time limit is set to off or reasonable amount. For me its 8hrs. After that if you can leave the run in the jupyter notebook inside the cloud desktop and access it anywhere. Its tricky to navigate the desktop in your phone but with few runs you would get used to it. Hope that answers your question.
Thank you for your reply, Debashish.
does it cost money to deactivate and activate a new machine?