I set up fastai on paperspace yesterday, and I want to share a couple of gotchas in case anyone runs into the same problem:
When select linux/ubuntu template, select version 16.04. i don’t believe fastai works with ubuntu v.18
You may get an error about “sudo rm /etc/apt/apt.conf.d/.” when you use the command curl files.fast.ai/setup/paperspace | bash
I ended up downloading the setup script from files.fast.ai/setup/paperspace (wget files.fast.ai/setup/paperspace) and comment out the line “sudo rm /etc/apt/apt.conf.d/.” before running it on the paperspace vm.
There are several resources on the fast.ai forums and GitHub that points too different machines to choose. For the most part, I think they’re all pointing to getting the fast.ai public template, but when it comes to choosing a particular machine, the voices seem split between choosing whichever you prefer (out of the M4000, P4000, and P5000) or just choosing the P4000 (links to the resources are attached below).
To the fast.ai alums still lurking on the forums, please let me know whichever you think is best from your experience!
This may be a little off topic (new to the forums, couldn’t figure out how to start a separate thread) but perhaps this will be useful to someone.
I recommend using Paperspace - it was a good alternative from AWS in Europe for both my personal projects and company work. However I kept running into a seemingly random ResourceExhaustedError on the GPU when training/retraining keras-based nasnet, xception & resnet models. This was on a dedicated P6000 instance, based on the template “ML-in-a-box” on Ubuntu 16.0, which comes with a desktop jupyter notebooks application.
I originally Paperspace’s support team pointed me to this known issue: https://github.com/tensorflow/tensorflow/issues/136. Later on it turned out that using the jupyter desktop app spawned but didn’t kill lots of processes in the background, leading to the GPU memory getting exhausted. The error in the notebook (when using the fit_generator() method) can be found at the bottom of this post.
So, in case you’re running into GPU resource exhausted errors, try using e.g. htop to see if there aren’t some unexpected processes eating them up.
Have a great day!
[[Node: training/Adam/gradients/dense_1/MatMul_grad/MatMul_1 = MatMul[T=DT_FLOAT, _class=["loc:@dense_1/MatMul"], transpose_a=true, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](flatten_1/Reshape, training/Adam/gradients/dense_1/Relu_grad/ReluGrad)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.```
For those reading setup help, please note as some have above that both FASTAI15 and FASTAI6GKZ are no longer active on Paperspace.
FASTAIGR45T may still be valid for $10 credit.
Please feel free to try that and you can also use a referral code, my referral M63NSKRhttps://www.paperspace.com/&R=M63NSKR or someone’s to get $10 credit.