Live coding 1

This topic is for discussion of the first live coding session.

Note that during these first few sessions we set up Paperspace from scratch. However we used this process to create a script to do it all automatically – it you’d rather use the script, here it is.

session 2 >>>

Links from the walk-thru

What was covered

  • How to setup and use a terminal/shell
  • What is a terminal? What is a shell? Why use them?
  • What’s a shell script? How to make and run one
  • Installing python the right way, and ensuring that there’s nothing in your system python messing things up
  • Using mamba/conda to install python libraries
  • Installing PyTorch (with GPU support if needed) and Jupyter
  • Running Jupyter

Video timeline - thank you @Daniel

00:00 - Intros

  • Nick (quant background) 1:01
  • Radek 2:22

03:26 - Jeremy’s way of doing stuff

04:48 - What is a terminal? What is a shell? Why use them?

  • Why use terminal?

06:07 - Setup your mind

  • Assume you only know basics of python and can use a computer but nothing else
  • So, your questions are extremely important to get things going

06:52 - Guests expectations

  • Radek speaks on motivations to learn paperspace
  • Nick speaks on too many options can be confusing and friction free is great 09:26
  • Aashi backs views of above

14:07 - How to setup and use a terminal/shell

  • feel free to ask questions anytime during the session
  • How to open a terminal
  • What is a terminal
  • How to install WSL for windows
  • How to get Ubuntu working as default
  • How to use a full screen for terminal

19:18 - Installing python the right way, and ensuring that there’s nothing in your system python messing things up

  • Try type python in terminal, see what you get
  • Why this is the python we don’t want to use?
  • How to find out where the python is running on your hard drive? which python
  • How to find out your current directory? pwd
  • Where do we want everything installed? your home directory

22:02 - Clean your home directory

  • How to find and install mambaforge?
  • How to make a directory? mkdir downloads
  • How to check inside a directory? ls
  • How to enter into a subdirectory? cd downloads with tab key to auto-complete
  • How to use wget to download? wget web-address (on my macpro I have to use sudo wget web-address)
  • How to paste text or web address into terminal? ctrl + shift + v (on my macpro I have to use cmd + v)
  • How to create another terminal to do other tasks? ctrl + t (on my macpro, I have to use cmd + t)
  • What does ~ mean? How about ~/Downloads?
  • How to move a folder/directory elsewhere? mv this-folder another-folder
  • How to uninstall ipython and jupyter? pip uninstall ipython jupyter
  • How to find the folder of ipython and jupyter and delete them from there? which ipython
  • How to delete a directory? rm -rf fold-name
  • We don’t want anaconda, ipython, jupyter in home directory

31:59 - Using mamba/conda to install python libraries

  • How do ls, ls -l and ls -l -h or ls -lh differ?
  • What is a .sh file? 33:25
  • How to read a file such as .sh in terminal? less file-name.sh
  • What signals the comments in the shell file or .sh file?
  • How to run the .sh script? bash Mambaforge-MacOSX-x86_64.sh (the linux one should work on mac too)? 34:00
  • What is a shell? What is this bash thing? and what does it do? (how to get terminal back to use bash? chsh -s /bin/bash)
  • What’s actually installing? does it only install python? 35:24
  • What does it mean to us to allow the installer to initialize mambaforge by calling conda init?
  • After the installing, reopen terminal, what does the newly appeared (base) mean?
  • How do we know we setup python correctly this time? which python
  • Why we must use a python in home directory which is different from the system python? 38:33
  • How to see hidden files? la -a (but only ls -ld .?* works on my Macpro) 39:55
  • What does a hidden file named .bashrc do? 40:13
  • How to edit this script? vim .bashrc
  • What to delete in the .bashrc to get us back from zero? meaning (base) is gone

40:52 How to automate the installation above
40:52 - What’s a shell script? How to make and run a script

  • go to fastsetup on github, find setup_conda.sh
  • where to get more info on using ls? ls --help (on my macpro terminal, I have to use man ls)
  • Why you should write a blog post on lectures by Jeremy and Radek 43:32
  • What does setup-conda.sh do? 46:10
  • How to download the sh file? 46:31 click raw, copy the link, and wget the-link to download (if not, download from raw manually)
  • How Jeremy explain the codes inside setup-conda.sh? 47:47
  • Why ./setup-conda.sh won’t work? and how to find out the problem? ls -l and no x means no permission to execute it 49:21
  • How to get back to the last command in the terminal?
  • How to grant a user the permission to execute a file? and what does this line of code say literally? chmod u+w setup-conda.sh 49:55
  • Which version of python does Jeremy use or recommend to use? 51:34
  • When does each version of python come to its end? Python | endoflife.date
  • Why recommend not to use the latest version of python?
  • How to get back to your home directory? cd

57:05 - Install other packages with mamba

  • What does mamba do for us about installing useful things? your own environment
  • How to install packages into python environment? 58:09
  • How to install python libraries into your python env? mamba, conda, pip
  • How conda and mamba differ? mamba is faster, but new so not all libs supported
  • Advice: only use conda, pip or mamba to install python libraries, most of time just use mamba
  • How to know more about mamba? mamba
  • How to install ipython with mamba? mamba install ipython
  • What does mamba ask and do for you? check which libs you need and ask for permission to install
  • How to close ipython or most programs on terminal? ctrl + d (or ctrl + z)

1:02:04 - Install pytorch

  • Where to find the right install command for pytorch? get-started-locally
  • What does this script mean mamba install pytorch torchvision torchaudio -c pytorch, in particular -c pytorch? 1:04:04
  • Will pip install mess up with my mamba kingdom? 1:05:06 yes, pip install does mess you up. So you should either pip uninstall or remove mambaforge folder and start install allover to be sure
  • How do we know pytorch is working? ipython and import torch and see

1:09:12 - Install Jupyterlab

  • How to install jupyterlab with mamba? mamba install jupyterlab

1:10:11 - Create the first notebook

  • how to create a nbs folder? mkdir nbs
  • go inside this folder cd nbs
  • run command jupyter lab
  • Where to find the link to open jupyter lab?
  • How to run jupyter lab without browser? jupyter lab --no-browser
  • How to find and check the content of .bash_history? cat .bash_history 1:12:53
  • How to make the most out of .bash_history file?
  • How to search the .bash_history commands? ctrl + r and use delete tab to clear the search, use esc tab to exit search
  • How to run a command starting with ju? !ju
  • How to run the last command? !!
  • How to show the last command? echo !!
  • How to add alias in command line? alias jl="jupyter lab"1:15:37
  • How to move around the command line, start of a line, end of a line, move a word at a time? ctrl + e, ctrl + a, opt + left or right (to move a word at a time, esc + B and esc + F can do with my macpro)
  • How to write and save alias for good? 1:17:05 open .bash_rc and add the alias and save (on my macpro, I have to use .bash_profile)
  • How to edit files in terminal without vim? use open . to open a folder, and to edit a file with vscode
  • Radek mentioned some additional advantages of WSL
  • How to create a jupyter notebook in a folder from jupyter lab 1:21:28
  • How to install jupyter ipywidget? mamba install ipywidgets
  • Can we do gui stuff with jupyterlab, like click and delete?

Additional tricks without going too far

  • how to clear the screen of terminal? ctrl + l
  • How to clear a line of command in terminal? ctrl + u
  • How to clear a word of a line of command in terminal? ctrl + w
19 Likes

Below is a crude but detailed summary of walkthru 1. Hope it can be some use to other beginners.
Please note:

  • everything is organised in question form,
  • and solutions/answers are immediately after the question mark ?
  • () includes solutions or answers only apply to my wierd macpro

Walkthrough 1

forum thread, walkthru playlist

00:00 Intros

  • Nick (quant background) 1:01

  • Radek 2:22

03:26 Jeremy’s way of doing stuff

04:48 Terminal

  • Why use terminal?

06:07 Setup your mind

  • Assume you only know basics of python and can use a computer but nothing else

  • So, your questions are extremely important to get things going

06:52 Guests expectations

  • Radek speaks on motivations to learn paperspace

  • Nick speaks on too many options can be confusing and friction free is great 09:26

  • Aashi backs views of above

14:07 Get terminal ready

  • feel free to ask questions anytime during the session

  • How to open a terminal

  • What is a terminal

  • How to install WSL for windows

  • How to get Ubuntu working as default

  • How to use a full screen for terminal

19:18 Check your python

  • Try type python in terminal, see what you get

  • Why this is the python we don’t want to use?

  • How to find out where the python is running on your hard drive? which python

  • How to find out your current directory? pwd

  • Where do we want everything installed? your home directory

22:02 Clean your home directory

  • How to find and install mambaforge?

  • How to make a directory? mkdir downloads

  • How to check inside a directory? ls

  • How to enter into a subdirectory? cd downloads with tab key to auto-complete

  • How to use wget to download? wget web-address (on my macpro I have to use sudo wget web-address)

  • How to paste text or web address into terminal? ctrl + shift + v (on my macpro I have to use cmd + v)

  • How to create another terminal to do other tasks? ctrl + t (on my macpro, I have to use cmd + t)

  • What does ~ mean? How about ~/Downloads?

  • How to move a folder/directory elsewhere? mv this-folder another-folder

  • How to uninstall ipython and jupyter? pip uninstall ipython jupyter

  • How to find the folder of ipython and jupyter and delete them from there? which ipython

  • How to delete a directory? rm -rf fold-name

  • We don’t want anaconda, ipython, jupyter in home directory

31:59 Install mamba

  • How do ls, ls -l and ls -l -h or ls -lh differ?

  • What is a .sh file? 33:25

  • How to read a file such as .sh in terminal? less file-name.sh

  • What signals the comments in the shell file or .sh file?

  • How to run the .sh script? bash Mambaforge-MacOSX-x86_64.sh (the linux one should work on mac too)? 34:00

  • What is a shell? What is this bash thing? and what does it do? (how to get terminal back to use bash? chsh -s /bin/bash)

  • What’s actually installing? does it only install python? 35:24

  • What does it mean to us to allow the installer to initialize mambaforge by calling conda init?

  • After the installing, reopen terminal, what does the newly appeared (base) mean?

  • How do we know we setup python correctly this time? which python

  • Why we must use a python in home directory which is different from the system python? 38:33

  • How to see hidden files? la -a (but only ls -ld .?* works on my Macpro) 39:55

  • What does a hidden file named .bashrc do? 40:13

  • How to edit this script? vim .bashrc

  • What to delete in the .bashrc to get us back from zero? meaning (base) is gone

40:52 How to automate the installation above

  • go to fastsetup on github, find setup_conda.sh

  • where to get more info on using ls? ls --help (on my macpro terminal, I have to use man ls)

  • Why you should write a blog post on lectures by Jeremy and Radek 43:32

  • What does setup-conda.sh do? 46:10

  • How to download the sh file? 46:31 click raw, copy the link, and wget the-link to download (if not, download from raw manually)

  • How Jeremy explain the codes inside setup-conda.sh? 47:47

  • Why ./setup-conda.sh won’t work? and how to find out the problem? ls -l and no x means no permission to execute it 49:21

  • How to get back to the last command in the terminal?

  • How to grant a user the permission to execute a file? and what does this line of code say literally? chmod u+w setup-conda.sh 49:55

  • Which version of python does Jeremy use or recommend to use? 51:34

  • When does each version of python come to its end? Python | endoflife.date

  • Why recommend not to use the latest version of python?

  • How to get back to your home directory? cd

57:05 Install other packages with mamba

  • What does mamba do for us about installing useful things? your own environment

  • How to install packages into python environment? 58:09

  • How to install python libraries into your python env? mamba, conda, pip

  • How conda and mamba differ? mamba is faster, but new so not all libs supported

  • Advice: only use conda, pip or mamba to install python libraries, most of time just use mamba

  • How to know more about mamba? mamba

  • How to install ipython with mamba? mamba install ipython

  • What does mamba ask and do for you? check which libs you need and ask for permission to install

  • How to close ipython or most programs on terminal? ctrl + d (or ctrl + z)

1:02:04 Install pytorch

  • Where to find the right install command for pytorch? get-started-locally

  • What does this script mean mamba install pytorch torchvision torchaudio -c pytorch, in particular -c pytorch? 1:04:04

  • Will pip install mess up with my mamba kingdom? 1:05:06 yes, pip install does mess you up. So you should either pip uninstall or remove mambaforge folder and start install allover to be sure

  • How do we know pytorch is working? ipython and import torch and see

1:09:12 Install Jupyterlab

  • How to install jupyterlab with mamba? mamba install jupyterlab

1:10:11 Create the first notebook

  • how to create a nbs folder? mkdir nbs

  • go inside this folder cd nbs

  • run command jupyter lab

  • Where to find the link to open jupyter lab?

  • How to run jupyter lab without browser? jupyter lab --no-browser

  • How to find and check the content of .bash_history? cat .bash_history 1:12:53

  • How to make the most out of .bash_history file?

  • How to search the .bash_history commands? ctrl + r and use delete tab to clear the search, use esc tab to exit search

  • How to run a command starting with ju? !ju

  • How to run the last command? !!

  • How to show the last command? echo !!

  • How to add alias in command line? alias jl="jupyter lab"1:15:37

  • How to move around the command line, start of a line, end of a line, move a word at a time? ctrl + e, ctrl + a, opt + left or right (to move a word at a time, esc + B and esc + F can do with my macpro)

  • How to write and save alias for good? 1:17:05 open .bash_rc and add the alias and save (on my macpro, I have to use .bash_profile)

  • How to edit files in terminal without vim? use open . to open a folder, and to edit a file with vscode

  • Radek mentioned some additional advantages of WSL

  • How to create a jupyter notebook in a folder from jupyter lab 1:21:28

  • How to install jupyter ipywidget? mamba install ipywidgets

  • Can we do gui stuff with jupyterlab, like click and delete?

Additional tricks without going too far

  • how to clear the screen of terminal? ctrl + l

  • How to clear a line of command in terminal? ctrl + u

  • How to clear a word of a line of command in terminal? ctrl + w

16 Likes

My expectations of walkthru 1 are very similar to @radek and nick:

I want to have Jeremy’s way of setting everything up and use Jeremy’s tricks to get things running without friction, and more importantly I can simply do the same setup online by opening a web-browser (like paperspace) without relying on a specific local machines with its own specifics.

So, here are my questions to @jeremy about the walkthru 1:

  • Why didn’t we apply this walkthrough directly to paperspace to setup python, mamba, jupyterlab and everything in the cloud rather than locally, as there are endless possibilities to mess up locally?
  • Will you show us how to do it on paperspace? or is it easy enough for us to apply what we learnt from walkthru 1 and directly apply it to paperspace?

I think doing all walkthrus in paperspace can really assure all of us that whatever Jeremy is demonstrating in the videos we can do the same online without too much trouble.

2 Likes

We’ll be doing paperspace soon. But I recommend having a working environment on your own machine too.

These walkthrus show you how to do things the best way, not the easiest way. And the best way is to be able to work directly on your own machine sometimes.

7 Likes

Creating a new environment.
My suggestion would be to create a new user on your system and use that to follow along. In that way if you have created scripts to setup an environment you can place them in a shared place for all users and then remove and recreate that user if it goes wrong, or make a ‘backup’ of the initial installation, although as things change that might not be useful. A new user is dependent on disc space that you have locally of course.

I note that users of linux based systems should be familiar with the available help systems these include man pages e.g ‘man some_command’ or info ‘info some_command’ indeed ‘info info’ is a good place to start.

The linux command ‘env’ lists all variable pointers available to the shell and this is where it can help to resolve issues. I note in my ‘condo’ system after I have activated it many ‘CONDA’ variables are defined, I haven’t got to install ‘mamba’ yet so not sure how or if they complement each other in the definition of variables that point to executables.

I hope this note does not complicate anything.

1 Like

I copied my descriptions into your timestamps - so it was very helpful thanks!

1 Like

Mamba 4.12.0 fails to install on older hardware.

I experienced an issue installing Mamba 4.12.0 latest release, the issue being that the embedded packages in the shell script file ’preconda.tar.bz2’ failed to extract, why, maybe because my hardware could becoming out dated, MacBook Pro mid 2009 OSX 10.11.6. However I managed to install Mamba 4.11.0. You may ask why install there as I have other more recent Apple and linux platforms, well simply because Anaconda was giving me problems with installing later releases of python there and would hang for long periods causing abortion of the script.

2 Likes

I’m trying to replicate everything in the Walk-thru. I use a local computer Ubuntu 20.04 that already has a conda environment. So I created a new user for the mambaforge installation. I switched to the new user as Jeremy does in the video, there is no file or folder in the home folder, no python too, but when I check jupyter, it is there:

  1. Why?
  2. Is it a problem?
    Thanks

Yes it’s a problem. Try which jupyter to see where it’s coming from. I’m guessing you’ve accidentally installed jupyter into your system python. Type sudo pip uninstall jupyter from your terminal with no conda env activated to remove it.

1 Like

Yes, you’re right, here it is:

/home/niyazi/anaconda3/bin/jupyter

I tried to uninstall but there is one more (weird)

which jupyter
/usr/bin/jupyter

Thank you, Jeremy

That’s another conda install, and shouldn’t be in the path of a new user. Something odd has happened there.

That’s in your system python.

So you need to uninstall whilst logged in as niyazi and with conda activated, and you also need to uninstall with no conda activated. You’ve sure installed jupyter a lot of times and places! :wink:

2 Likes

Another wsl tip - in Windows and explorer \\wsl$ will show all of your wsl instances and you can drill into them.

3 Likes

Hi everyone! I wrote a short guide on how to install mambaforge based on Jeremy’s walk-thru 1. Hope you find it useful :slight_smile:

11 Likes

This is exceptionally well done - thank you for your hard work on it!

4 Likes

Here are my notes on Walkthru 1, there are two parts, first part is about the tips and shortcuts mentioned by Jeremy and the second is my terminal timeline. It may be handy when you are watching it.

Tips & Shortcuts from Walkthru 1 :

  • To move things, use mv, the first is the source and the second is the target, if there is no target then it works like a renaming, here is an example:
mv miniconda miniconda.old
  • To remove files & Folders : remove recursively force r recursive and f forced.
rm -rf 
  • Some list (ls) options:
ls -l # -l is long-form ls
ls -h #h human-readable form, for example, file size as MB or GB but not bytes.

# When you need to use multiple flags then no need to put two times
ls -lf

ls -a # for listing all files (hidden)
  • Aliases
# Aliases

# up and down arrows work as shortcuts because of .bash_history

# it is handy to create your scripts by copying from the history.

# CTRL-r is great for searching the history in the terminal

# also !ju runs the last one starts with 'ju'

# !! runs the last command

alias jl="jupyter lab" # no spaces and if you want make the aliases persistent then
# to them put in the .bashrc

My command timeline for the installation with some comments.

  • Do not use the python that comes with the system installation.

  • Download mambaforge and work install it and work on the ‘base’ environment

# .sh file is a script and to see it
less Mambaforge-Linux-x86_64.sh

# to install
bash Mambaforge-Linux-x86_64.sh

# remove the mamba for a automatical setup 
rm -rf mambaforge/

.bashrc #  automatically runs when you run the terminal so,

# edit the file for removing the conda initialisation, just delete the conda part.

# that will remove the [base] from the command line

# fastsetup arranges everything for a new computer

https://raw.githubusercontent.com/fastai/fastsetup/master/setup-conda.sh

./setup-conda.sh #no need for bash at this time because it is already in the script's first line

# doesn't help on the first try, because We need to change permission for execution

chmod u+x setup-conda.sh

# then again:

./setup-conda.sh

# in my case I've got this error

# ./setup-conda.sh: line 22: curl: command not found

# and I install curl: 
sudo apt install curl

# then pytorch from here: https://pytorch.org/get-started/locally/

mamba install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

# -c pytorch means install things from pytorch channel

# then Jupyter or JupyterLab

mamba install jupyterlab
2 Likes

I am still catching up with the walk thrus but here are some questions:

  1. Did anyone have an issue with ‘—no-progress-meter’ in the setup shell script from the repo? I removed the option and the script ran fine but I’m wondering what’s different about my setup that doesn’t recognise the flag.

  2. Is there any reason why we are installing Jupyter Lab and not Jupyter Notebook?

2 Likes

Same here, removed ‘–no-progress-meter’ and it worked fine after that…

1 Like

I do not know the reason, but you can install Jupyter-Notebook too. Both work well in the same system.

I guess that must only be in some more recent version. What OS are you using?

Hi Jeremy,

What are you entering to delete the section for this section?

  • What to delete in the .bashrc to get us back from zero? meaning (base) is gone

You have highlighted the section at the end but I don’t know what you are pressing to make it gone.

Thank you!!!