Paperspace setup help

Never mind about lacking a console. On the browser tab showing the directory, the user can choose ‘New’ and then ‘Console’. I just needed to white list the website, i.e., localhost:88xx or whatever.

Yikes! I set up a paperspace account and created the suggested machine using the fastai template tonight. Got the email an hour and a half ago that it was ready, but have spent most of that time with the machine crashing and spinning its wheel trying to start up. With considerable time and effort I can get it to run, launch jupyter notebook and execute a few cells, then I lose the connection, then the machine locks up, and its back to the game of trying to restart the machine.

I’ll try again tomorrow, but paperspace is not looking like a service that should be recommended.

Hmmm…I’ve had a completely different experience. Quite reliable, actually. Paperspace runs out of P5000, so I end up on a GPU+, but it doesn’t crash, etc.

FWIW, I connect to their east coast servers and use the Gradient / fastai template.

It sounds like the Gradient approach works better than the VM approach. I may try that if the VM remains unstable. I tried the clouderizer to collab approach last night, but after 20 minutes loading packages for one notebook into collab, I got an error. The notebook never ran.

I’ve been using colab, too, but not with clouderizer. The plain-vanilla notebooks it creates have numpy, pandas, sklearn, matplotlib and display (for IPython.display) modules available by default. If you use one, you will need to install fastai, but 20 minutes seems excessive. Mine’s been more like 5 at the most. I used
!{sys.executable} -m pip install fastai.

And as mentioned, colab uses K80 GPUs, so calculations are s-l-o-w. It is free, though.

Also, if you use Gradient: if you restart a notebook, it won’t know where to look for fastai. You will need to add the code below to use the fastai functions.

import sys
sys.path.append('../../')

Never mind about being s-l-o-w. Apparently, we can use Google’s TPUs now.

The picture shows an arrow pointing to the ‘Connect’ button, but the setting to change to a TPU can be found under Edit>Notebook Settings>Hardware Accelerator

I was able to get the first lesson working on Google Collab, up to the point of needing the data files. It took several minutes to load all the fastai packages. Then when I got to the point of needing the data files and saw what it took to link to Google Docs, I decided it wasn’t worth using collab. Collab also disconnects after a short time. I have limited time that I’d rather spend getting work done than working around Collab’s limitations. Still, if you have the time and can’t afford another solution, Collab can work.

I also tried a Paperspace gradient notebook. This seems easier at least initially, but its not clear to me how to update from the repo or get other data into the notebook. The fact that you can launch the same notebook on a cheap non-GPU machine is also nice. I’d say this is a viable approach, assuming you can get new data loaded and update the fastai material. One issue, at least for me, is that when I picked east coast, it only allowed me to use a higher cost GPU. However, the training took quite a bit of time with that machine, so the better machine is probably the right choice.

Today I will try using SSH to the Paperspace Machine. I think I’d be most comfortable with a traditional unix command line and file system–that’s what I’m used to when running Jupyter on my local machine. As long as it works, I’d rather pay $.51/hour than spend hours on workarounds.

Not sure how many are using local eGPUs for the courses. I have a Titan XP in an eGPU enclosure on Mac. I got this courtesy of the NVIDIA GPU grant program for research I’m doing that doesn’t suffer from the low external data transfer rates (all the data is transferred, a long GPU computation runs and then only a few results are returned). However, it can be slow if there are frequent data transfers between the eGPU and the Mac. It is also a constant struggle to keep patching the Mac to access the eGPU, since apple doesn’t support NVIDIA eGPUs natively.

1 Like

Another update. Following the instructions here:

I was able to ssh into my Paperspace machine using ssh paperspace from my local machine. I could then launch Jupyter notebook on the paperspace machine and connect to it locally using http://localhost:8888/

However, the notebook server shows my local files, not the paperspace machine files. Any suggestions for how to fix this?

Also, how can I avoid a new token every time I want to connect to the remote server?

Apparently, when I had a local Jupyter Notebook server running that was preventing me from accessing the remote server. I thought I had that shutdown, but after rebooting I was able to connect to Jupyter on the remote machine and see the remote filespace.

Also, so far using SSH and connecting as per the instructions in the link above, I’ve not had the remote paperspace machine freeze. As others have noted, the paperspaces web terminal seems to be the issue. My advice for using paperspace machines is to start them up without trying to start the web terminal: Log in to your account and then click Core, Compute, Machines from the left menu, then switch the view of your machines to the list view by clicking the faint lines to the right of Machine at the bottom of this image:

image

Now you can click on the machine name and click the Restart button. This will start your machine without trying to launch the terminal in your browser. After that SSH to your machine, launch Jupyter Notebook, and connect from your local web browser.

I spoke too soon, my Paperspace machine shut itself down part way through training. I tried running locally on my Mac only to discover the missing cuda90 issue. Now I’m off to look for another platform to run the notebooks on. I feel like I’m playing Whac-a-Mole and the moles are winning :wink:

I had a similar experience, and was only partially reimbursed for the 64 hours the machine was running before I found out and shut it down. Comments on the thread here led me to believe it might take hours to provision the server, and nothing in the setup or the email I received from Paperspace mentioned that the server would be automatically turned on after provisioning and that the hourly billing would begin. So much for the “free” class in Deep Learning. Very disappointing.

2 Quick questions folks,

1 - should there be some sort of graphic display of the desktop through the paperspace web client like in the setup video?

2 - is the fast.ai image up to date? I only see ml1 and dl1 courses in there, shouldn’t there be a second one?

Hi Amy-
I filed a complaint with the Better Business Bureau:
https://www.bbb.org/us/ny/brooklyn/profile/information-technology-services/paperspace-0121-170390
I also wrote to someone at fast.ai - sorry I can’t find the email. I’m not sure what did it - but I agreed to resolve the issue with the BBB if was refunded. Their billing practices are faulty. There needs to be handshake before the billing starts.
Good luck - don’t get discouraged.

1 Like

I am getting frustrated with attempts to work on remote machines. I believe I set up everything correctly this time per the instructions given here: https://github.com/reshamas/fastai_deeplearn_part1/blob/master/tools/paperspace.md
Then I loaded the Jupyter Notebook. I received some error message about expecting a string and receiving a number, or vice versa. I thought that was odd. The notebook loaded anyway. And it appeared to be working fine. Until I realized that there were only 5 executable cells.

I tried reloading the notebook multiple times. No error messages this time, but only 5 cells. I did a git pull for the second time. Still only five cells. I restarted the machine. Still only 5 cells.

So I went to lesson 2. And got the same error: " The error was:

[sprintf] expecting number but found string

See the error console for details."

Here is what the console displays:

Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
    http://localhost:8888/?token=557215ff2d4e247928bea3764848a57b70de6b9e273c7b08

[W 17:25:45.617 NotebookApp] Forbidden
[W 17:25:45.617 NotebookApp] 403 GET /api/sessions?=1538688180184 (73.231.244.157) 2.00ms referer=http://184.105.191.13:8888/tree/courses/dl1
[W 17:25:45.618 NotebookApp] Forbidden
[W 17:25:45.619 NotebookApp] 403 GET /api/terminals?
=1538688180185 (73.231.244.157) 0.76ms referer=http://184.105.191.13:8888/tree/courses/dl1
[W 17:26:02.436 NotebookApp] Forbidden
[W 17:26:02.436 NotebookApp] 403 GET /api/sessions?=1538688180186 (73.231.244.157) 1.37ms referer=http://184.105.191.13:8888/tree/courses/dl1
[W 17:26:02.440 NotebookApp] Forbidden
[W 17:26:02.441 NotebookApp] 403 GET /api/terminals?
=1538688180187 (73.231.244.157) 0.65ms referer=http://184.105.191.13:8888/tree/courses/dl1
[I 17:26:02.457 NotebookApp] 302 GET /?token=557215ff2d4e247928bea3764848a57b70de6b9e273c7b08
(73.231.244.157) 0.38ms
[I 17:26:12.583 NotebookApp] Kernel started: a95b78b8-b002-4486-ad4b-88d355be1987
[I 17:26:13.108 NotebookApp] Adapting to protocol v5.1 for kernel a95b78b8-b002-4486-ad4b-88d355be1987
[W 17:32:24.569 NotebookApp] Notebook courses/dl1/lesson2-image_models.ipynb is not trusted
[I 17:32:25.068 NotebookApp] Kernel started: 191373b1-be4c-4901-8ac8-f9532592269e
[I 17:32:25.611 NotebookApp] Adapting to protocol v5.1 for kernel 191373b1-be4c-4901-8ac8-f9532592269e

Could someone please advise?

The only unusual occurrence I can think of is when I first tried to copy the IP Address, I clicked on it, so the console tried to load it from the same page, thereby creating an interruption. @dillon @jeremy

I tried getting a new machine. I reinstalled everything. I waited to do conda env update. Before and after I got the same result. The only thing consistent is the IP Address from before. @dillon

As a followup to help others, my Paperspace VM was shutting down due to the 1 hour timeout. If you SSH to the VM and start Jupyter Notebook, then access your Jupyter Notebook server from your local machine, Paperspace will not see your interaction. I bumped the timeout up to 8 hours and have not had trouble since.

I set up fastai on paperspace yesterday, and I want to share a couple of gotchas in case anyone runs into the same problem:

  1. When select linux/ubuntu template, select version 16.04. i don’t believe fastai works with ubuntu v.18
  2. You may get an error about “sudo rm /etc/apt/apt.conf.d/.” when you use the command curl files.fast.ai/setup/paperspace | bash
    I ended up downloading the setup script from files.fast.ai/setup/paperspace (wget files.fast.ai/setup/paperspace) and comment out the line “sudo rm /etc/apt/apt.conf.d/.” before running it on the paperspace vm.

Which template should I choose?

There are several resources on the fast.ai forums and GitHub that points too different machines to choose. For the most part, I think they’re all pointing to getting the fast.ai public template, but when it comes to choosing a particular machine, the voices seem split between choosing whichever you prefer (out of the M4000, P4000, and P5000) or just choosing the P4000 (links to the resources are attached below).
To the fast.ai alums still lurking on the forums, please let me know whichever you think is best from your experience!



Reshama Shaikh - https://github.com/reshamas/fastai_deeplearn_part1/blob/master/tools/paperspace.md
tcvieira - https://gist.github.com/tcvieira/d29d38068a6cd2c455baaaf0d183534b

Hi,

I’m not an expert, but personally I’ve used the P4000 and it worked just fine with all notebooks of Part 1.

So, I think the P4000 is a good option, you will be able to learn efficiently all the material in the course!

Hello Antoine,

Thanks for the advice! I can’t wait to start messing around with the coursework.