During this session we create a pre-run.sh file which has a bug in it. To fix the bug, you need to add the following line to the end of it:
cd
If you don’t add that line, you won’t be able to see your existing notebooks any more! See the start of the next video for details.
More importantly: since we recorded this, we’ve now automated the setup process, so you can skip nearly everything in this session! To set up your paperspace environment, follow the instructions here:
04:13 - Create an empty notebook and symlink from persistence storage
Why open a new window to use jupyter lab? to keep paperspace interface for shutting down when finished
Why should you read paperspace docs? What did Jeremy find out? 05:16
What are the interesting folders inside the root directory?
How is storage folder different from notebooks directory? Why both of them exist for good reason? 06:32
Should we worry about using pip install when paperspace uses conda a lot? 08:02
How to pip upgrade packages into the home directory with --user? pip install -U --user fastcore13:14
What folder we want to be there next time when we open notebook? .local/
How to save this .local/ into persistence storage? mv .local /storage/
How to delete/clear everything left to the cursor? ctrl + u
How to delete/clear everything right to the cursor? ctrl + k
But it is better to save .local to /storage/cfg, how to do it in the lazy way? mkdir /storage/cfg; mv .local !$
How to create a symlink from the persistence storage to this notebook? ln -s /storage/cfg/.local/ 16:04
How to check whether this symlink is created? ls -la to see the changes
when create a new notebook, don’t forget to choose the fastai image and choose advance option to remove the git repo for a clean notebook 18:40
19:24 - Create pre-run.sh from scratch to automate .local symlink from storage
How to create a python file to setup the symlink first before running jupyter lab so that when we do pip install -U --user the packages will automatically link to storage/cfg/.local? 19:24
Does Jeremy think paperspace is the way to have easy to use GPU for fastai in cloud? 21:32 yes
How to make a symlink from /storage/ back to this notebook’s /notebooks directory? cd /notebooks/; ln -s /storage/
What does this step above do? to link the /storage/ folder back into /notebooks/ folder
Why to access the /storage/ folder inside notebooks is useful? so that we can create and edit new files inside notebooks with jupyter lab
How to create a text file, edit it and save it inside /storage/ with jupyter lab? 24:53
What shall we put inside this text file?
#!/urs/bin/env bash: run the following script in bash
cd: go to the home directory
ln -s /storage/cfg/.local/: symlink the persistence storage .local folder to this notebook’s home directory
How to find where the bash is? which bash may give us /bin/bash
Which name should we name this text file? pre-run.sh
Why we can’t run ./pre-run.sh directly? 27:57 execution permission is needed, ls -la can show us the permission status
What is the usual way of setting execution permission? chmod u+x pre-run.sh
How Jeremy set permission? chmod 744 pre-run.sh and what does 7, 4, 4 mean each? 29:47
How to check the status again and run the .sh file again? ls -la; ./pre-run.sh;
How to ensure there won’t be any .local folder inside a notebook when it opens next time? add rm -rf .local into pre-run.sh, so now the file looks like below
#!/urs/bin/env bash
cd
rm -rf .local
ln -s /storage/cfg/.local
Let’s create a new notebook to see whether it has the .local linked from the storage? 32:28
33:15 Create SSH keys from scratch and automate .ssh symlink from storage with pre-run.sh
First, how to create a .ssh folder to save keys? 33:15mkdir .ssh
How to upload the keys from local computer to the notebook with gui? loaded into /storage/
How to move them back into ./.ssh folder? cd .ssh; mv /storage/id_rsa* ./
How to check whether the SSH keys are in proper permission? ls -la
How to only allow user to have all the rights to the entire .ssh directory ? 34:24chmod 700 .
How to allow user to only read and write on the private key? chmod 600 id_rsa
How to allow user to read and write and everyone else only read the public key? chmod 644 id_rsa.pub
How to connect the notebook with github using the ssh key? inside .ssh directory, run ssh git@github.com and type yes to continue to see the connection successfully
How to see the connection process in detail? ssh -vvv git@github.com
How to symlink .ssh folder from /storage/cfg/.ssh? 36:44
Jeremy I am not sure of the validity of what you do with you ssh keys. The concept of root is a little confusing here, on an individuals computer the concept is pretty straight forward the individual has root permissions on a remote server who actually is root the individuals maintaining that server environment have root privileges on all the servers in that environment I believe. I flag this here as i am not a professional System Administrator, will try to look into further but the advice of a SysAdmin person at U may be prudent.
It’s Elizabeth II’s 70th Jubillee celebrations here so I am honor bound to have a few drinks and party.
I got things working as planned and as taught in the lesson. But I’ve noticed that since creating the pre-run.sh file (which is persistent across servers) that my other servers are bit screwed up now.
I had previously created two other Paperspace servers using git clone of the fastai repo, one directly from the fastai repo and one from a fork on my own github account. Those notebooks are now not showing up as files in Jupyter Lab on any server. For example, here is what the Gradient server looks like prior to starting the virutal machine:
I recall that Paperspace opens Jupyter in the /notebooks directory by default. If enter !pwd I can see that I am in /root. I found this way using cell magic to change the working directory:
Can I change the default directory back to notebooks?
P.S. I entertained the idea that it had something to do with the symlink that we created to /storage in /notebooks last night, but as Jerermy said (and I have confirmed), the /notebook drive does not persist across servers.
Thanks for pointing this out. I think it can be useful in situations when you sort of work out a command using non-destructive commands and then want to give it a ‘destructive’ command.
As in the example below, I’m using ‘ls’ to see if the file removeme.txt is there, then I just take that argument to ls command and pass it to rm command to get rid of it. The last ‘ls’ checks to see if it’s gone by using the argument to the last command which happens to be the ‘rm’ command.
But honestly though, I try to stay away from these fancy globbing things when doing anything like rm because if I happen to forget, the results can be catastrophic.
(fastai2022) mikemoloch@moloch titanic % touch removeme.txt
(fastai2022) mikemoloch@moloch titanic % ls removeme*
removeme.txt
(fastai2022) mikemoloch@moloch titanic % rm !$
rm removeme*
(fastai2022) mikemoloch@moloch titanic % ls !$
ls removeme*
zsh: no matches found: removeme*
I created access by making a symlink in home directory to /notebooks. I suppose I could add this to pre-run.sh and it will be automatic at boot. Is this the best way. Would still like to know why this happened.
This is the stuff that drives me crazy and consumes a lot of time!
If we deleted the mamabaforge directory for a fresh install as Jeremy suggests, then will this also remove the pip installed modules, which are stored in /storage/.local? I would expect we’d have to delete those as well?
Not sure what I am saying myself as the field of security is so complex and even if there is a problem. But my gut feel is storing a private key on a cloud server is a security risk. On your client box you are the admin of the whole system. On a cloud server you are admin of your slice of the cloud but there are admin at a level above you that have greater privileges than yourself on the cloud server. Any way still not sure what I am saying here. I did this search
I would have thought that to access the git hub server for git clone etc would be to repeat what you did on your client box to access git server direct, but think that impracticable because each time you login to the cloud you would never be on the same server as last time as the physical address of the sender/cloud_client would be different. I am sorry this gets so complicated to explain if your not a security expert which I am not. Anyway I just feel there may be securer ways to do remote calling from a cloud server. I appreciate this is off topic from the main gist of these walk throughs perhaps should be in a separate discussion or not if this is just pie in the sky
We discussed that a bit in the walk-thru. There are 3 levels of security you can choose from:
Use a separate GitHub Personal Access Token (PAT) for the machine. This is most secure, since you can define the exact permissions it has
Use a separate SSH key pair for the machine. This is less secure, since it has access to your whole GitHub account
Use a single SSH key pair across all your machines. This is the least secure, since it has access to your GitHub account and any machines you have that are accessible over SSH.
Hopefully that helps explain your options a bit better.
Here is a rough but detailed note which may help search in the video.
Walkthrough 4
forum thread
00:00 Create a total empty notebook
How to be lazy and a great programmer? 03:02
04:13 Create an empty notebook and symlink from persistence storage
Why open a new window to use jupyter lab? to keep paperspace interface for shutting down when finished
Why should you read paperspace docs? What did Jeremy find out? 05:16
What are the interesting folders inside the root directory?
How is storage folder different from notebooks directory? Why both of them exist for good reason? 06:32
Should we worry about using pip install when paperspace uses conda a lot? 08:02
How to pip upgrade packages into the home directory with --user? pip install -U --user fastcore 13:14
What folder we want to be there next time when we open notebook? .local/
How to save this .local/ into persistence storage? mv .local /storage/
How to delete/clear everything left to the cursor? ctrl + u
How to delete/clear everything right to the cursor? ctrl + k
But it is better to save .local to /storage/cfg, how to do it in the lazy way? mkdir /storage/cfg; mv .local !$
How to create a symlink from the persistence storage to this notebook? ln -s /storage/cfg/.local/ 16:04
How to check whether this symlink is created? ls -la to see the changes
when create a new notebook, don’t forget to choose the fastai image and choose advance option to remove the git repo for a clean notebook 18:40
19:24 Create pre-run.sh from scratch to automate .local symlink from storage
How to create a python file to setup the symlink first before running jupyter lab so that when we do pip install -U --user the packages will automatically link to storage/cfg/.local? 19:24
Does Jeremy think paperspace is the way to have easy to use GPU for fastai in cloud? 21:32 yes
How to make a symlink from /storage/ back to this notebook’s /notebooks directory? cd /notebooks/; ln -s /storage/
What does this step above do? to link the /storage/ folder back into /notebooks/ folder
Why to access the /storage/ folder inside notebooks is useful? so that we can create and edit new files inside notebooks with jupyter lab
How to create a text file, edit it and save it inside /storage/ with jupyter lab? 24:53
What shall we put inside this text file?
#!/urs/bin/env bash: run the following script in bash
cd: go to the home directory
ln -s /storage/cfg/.local/: symlink the persistence storage .local folder to this notebook’s home directory
How to find where the bash is? which bash may give us /bin/bash
Which name should we name this text file? pre-run.sh
Why we can’t run ./pre-run.sh directly? 27:57 execution permission is needed, ls -la can show us the permission status
What is the usual way of setting execution permission? chmod u+x pre-run.sh
How Jeremy set permission? chmod 744 pre-run.sh and what does 7, 4, 4 mean each? 29:47
How to check the status again and run the .sh file again? ls -la; ./pre-run.sh;
How to ensure there won’t be any .local folder inside a notebook when it opens next time? add rm -rf .local into pre-run.sh, so now the file looks like below
#!/urs/bin/env bash
cd
rm -rf .local
ln -s /storage/cfg/.local
Let’s create a new notebook to see whether it has the .local linked from the storage? 32:28
33:15 Create SSH keys from scratch and automate .ssh symlink from storage with pre-run.sh
First, how to create a .ssh folder to save keys? 33:15 mkdir .ssh
How to upload the keys from local computer to the notebook with gui? loaded into /storage/
How to move them back into ./.ssh folder? cd .ssh; mv /storage/id_rsa* ./
How to check whether the SSH keys are in proper permission? ls -la
How to only allow user to have all the rights to the entire .ssh directory ? 34:24 chmod 700 .
How to allow user to only read and write on the private key? chmod 600 id_rsa
How to allow user to read and write and everyone else only read the public key? chmod 644 id_rsa.pub
How to connect the notebook with github using the ssh key? inside .ssh directory, run ssh git@github.com and type yes to continue to see the connection successfully
How to see the connection process in detail? ssh -vvv git@github.com
How to symlink .ssh folder from /storage/cfg/.ssh? 36:44
On free subscriptions, notebooks with free machines will always be set to public. Upgrade to Pro/Growth if you would like to set free machines to private.*
I do not know what that means exactly and if it was a security risk considering some of us may have put a private key on the server.
any idea?
No chance to google these ones though 
Watching the walk-thru our only recourse!
