This topic is for discussion of the 3rd live coding session.
- Creating and using a conda environment
- How conda environments are actually implemented
- Creating a Paperspace notebook
- Installing pip packages into your home directory
- Persistent storage and mounted drives
- Opening JupyterLab in Paperspace
- Copying SSH keys to Paperspace
Video timeline - thank you @Daniel
00:00 - Catch-up Questions from the last session
- Why does Jeremy prefer to have just one base environment?
- What advantages of having just one base environment and deleting it and creating a new and updated one when needed?
- When does Jeremy occasionally need to create a separate environment?
- Why this way is more user-friendly?
- Should you install fastai or simply fastbook?
settings.ini and fastbook setup (more advanced)
- What’s installed when installing fastboot?
- Why was sentensepiece a problem? because it was not in
08:19 - PATH variable
09:09 - why python comes from the specific directory?
- What is a path or environment variable?
- How to print out the content of the path variable?
- Is this a colon-separated string?
- What is this long string or multiple strings? addresses of many directories with different packages/libraries stored inside, including the python we are using
- Why is the PATH variable so important? As long as the python library is stored in one of the addressed listed inside the Path variable, we can just type
pythonto use it
- How to run
which pythonwithout typing
pythonwhen the last command was
- What’s inside the
- is the
mambaforgedirectory very similar to mac/linux root directory?
- How libraries,
PATHvariable work together? 11:54
12:22 - Create a new environment
- How to create a different environment?
- How to create an env named
tmpwith python<3.10 and fastcore inside?
mamba create -n tmp 'python<3.10' fastcore
- How to activate/inactivate an env?
mamba activate tmpor
- Where does the python come from after you enter this new tmp environment and run
- What is inside the newly generated folder
- Does this python come from the python libraries stored inside
envs/tmp/libcontain all the libraries of this new environment?
- What are hard links? Why it is very nifty? and symlink later 15:30
- How to quickly move directly from
- Why does Jeremy recommend not freezing particular versions of libraries?
18:27 - Paperspace gradient notebook
- Why nice to have things set up locally?
- Can we actually do a lot of fastai things without GPU? make predictions, play with small datasets, and others
- Why paperspace when we need a GPU? What are paperspace gradient notebooks?
- Why the pro account is a good deal?
- Why persistence storage is important? Why each gradient notebook is like a separate server or new computer?
21:19 - How to create a notebook from scratch?
- How to pick on the GPUs for your notebook?
- Why select the auto-shutdown?
- How to share your notebook?
- What is the workspace url? How to fork the fastbook repo through this gradient notebook? #question
- Why to keep the paperspace gradient interface paper on? easy GPU shut down
- Why copy and paste another paperspace gradient page link?
- How to switch to Jupyter notebook interface for the gradient notebook? 25:52
- How to get used to Jupyter lab? 28:39
- How to maximize the notebook window?
- How to switch between tabs?
ctrl + shift + [ or ]
- How to turn on and off the side bar?
ctrl + b
- How to remove the status bar? 29:47
33:12 - The python debugger
- How to do the notebook debugger?
- How to use the non-graphic debugger? 33:12 add
%%debugat the top of a cell you want to debug?
- How to get help with
ipdb? just type and run
- Why not use notebook debug with non-graphic debugger together?
- Why and how to use the
%%debugwith a function?
- How to run the next line of code with debugger?
- How to print a content of
- How to repeat the last command? just press enter
- How to check the file and where the execution line is?
- How to find out which function made the call of the current line?
- Why the non-graphic debugger is great by Radek?
- How to use
set_traceto debug at a specific place of a function? 39:10
- How to exit the debugger?
- Why the python debugger is robust and good time investment?
43:08 Install or Upgrade env for the notebook
43:08 - Installing pip packages into your home directory
- If you
pip/conda installin the terminal, will you have them next time when open this notebook?
- What is and where is the persistence storage? from root directory
- How to check the storage spaces paperspace provide you?
- What is the datasets storage? 45:01
- Where do we want to install libraries to? storage folder
- Which kind of libraries we are safe to use pip to install? python libraries, even with GPU, but not pytorch nor cuda
- Do we have access to the root directory of gradient notebooks?
- How to update libraries into root/storage directory?
pip install -U --user fastcore, thanks to
- Is the access to root directory problem solved by
pip install --user?
- Is this the reason why Jeremy prefers
- How to check the version of a library like
import fastcore; fastcore.__version__;
49:21 - Persistent storage, mounted drives, and symlinks
- Where does
pip install --userstore libraries? 49:21
ls -lato see
.localfolder, and check
- How to make sure
.localis still there when we use the notebook next time? 50:02
- How to update
.localto persistent storage without take much space? symlink
- How to create a symlink from
ln -s /storage/.local
- How do you know the symlink is created successfully? check the
.localagain to see
.local => storage/.local
- How does Jeremy make sure every time he opens a notebook everything is already set up for him?
- Which file of root will run to set up everything for us when start a notebook?
- What’s the content of Jeremy’s
- How do we create our own
/storage/.bash.local? or do we just use Jeremy’s? #question
- Why we don’t save stuff in the paperspace’s root but use symlink? There is not real root there, things will be gone next time you enter
- Can you even symlink SSH key, kaggle key?
56:27 - Paperspace have different python environments by default
- How Jeremy solve the fastcore problem in real time? 56:27
- How does python find modules or path for libraries? 1:00:30
- Where are the places python will search for libraries?
import sys; sys.path;
- How to find
fastcoreinside a jupyter notebook (still not working inside python)?
import fastcore; fastcore?;
import sys; sys.path;give us the same search addresses in python vs ipython environments? No
- Finally, what is the problem? the fastcore is installed in a python3.7 env not the python3.9 env
- How to use python3.7?
- But why ipython is running python3.7?
- How safe it is to upload SSH keys into paperspace? 1:08:15
1:09:34 - Creating a Paperspace notebook with everything set up automatically
- How to create a server from scratch? 1:09:34
- When Jeremy creates a new notebook is the
gitdirectory available automatically? yes, but you need to start a terminal to let it appear 1:12:13
ls; ls /notebooks/;
- Why only open terminal then we can see
gitdirectory? maybe because Jeremy’s setup is done by
/storage/.bash.localwhich is executed in terminal
- Which file does paperspace run to setup everything for us?
run.shwhich has a
pre-run.shwhich maybe we should use it instead of
- How to read the end of a file in terminal?
- How to turn
mv /storage/.bash.local pre-run.sh
- Why paperspace is what Jeremy want to use? paperspace listened and make it the way Jeremy want it to be
- How to create a new notebook from scratch step by step? 1:15:01 watch this carefully
- How should we use workspace url and what is it for? if you give it your github repo url, the stuff will be ready at
/notebooks/directory when you open this new notebook
1:16:35 - Copying SSH keys to Paperspace to communicate with github
- How to setup SSH on paperspace from scratch? 1:16:35
- Why do we create SSH for the notebook? How to create a SSH in paperspace terminal?
- How to interpret the files and directories created by
ssh-keygen? nicely explained
- How to upload your local private and public keys to the
- How to show all the files starting with
mv /notebooks/git/id_rsa + tabto show them all
- How to move them all to the current directory?
mv /notebooks/git/id_rsa* ./
- How to change the permission to make sure no one can read private key?
chmod ag-r id_rsa
- How to use SSH key to ask github to connect with this notebook?