Live coding 6

Part 1 2022
#63

Thanks for your answer. It’s help a lot.

#64

Hi @galopy,

I also noticed that paperspace has mamba now pre-installed. I checked out your blog (awesome!) and found that I have set things up almost identical.

However, when trying to install any other (python based) package through mamba (so not universal-ctags) Mamba still wants to install way too much stuff, e.g. also a python version together with a huge number of other stuffs.

It looks as if:

  • the system is using python 3.9.13 in /usr/local/bin/python (–> which python and python --version in the terminal)
  • all main dependencies such as torch and fastai are all installed in usr/local/lib/python3.9/dist-packages/ (run ipython from the terminal and import for example fastai and then run ?fastai)
  • this is the stuff that is used by ipython and jupyter lab
  • installing python stuff with pip -U --user is integrated with that (I tested it with a couple of libraries)
  • mamba is a totally separate installation with a separate python (version 3.10.6) installed in /root/mambaforge/bin (run mamba list in the terminal)
  • this mamba is and can only be used to install (mamba install -p ~/conda <package>) none python specific stuff such as unverisal-ctags, for anything else, it will try to install a completely fresh installation of python as well… and such packages will also not be integrated with the system python (usr/local/bin/python)

Long story short, we should still install any python specific additional libs with pip for it to work in any jupyter notebook.

Another way to see this, is by creating a notebook (an actual notebook…) and run import sys; sys.path which doesn’t mention anything on mambaforge:

['/notebooks',
 '/usr/lib/python39.zip',
 '/usr/lib/python3.9',
 '/usr/lib/python3.9/lib-dynload',
 '',
 '/root/.local/lib/python3.9/site-packages',
 '/usr/local/lib/python3.9/dist-packages',
 '/usr/lib/python3/dist-packages']

Is this also your experience? Or am I doing something wrong?

#66

Hi,

Thank you for reading my blog. I really appreciate it.
I also get the same output as yours. I got this: (starting from '/notebooks/paddy' directory)

['/notebooks/paddy',
 '/usr/lib/python39.zip',
 '/usr/lib/python3.9',
 '/usr/lib/python3.9/lib-dynload',
 '',
 '/root/.local/lib/python3.9/site-packages',
 '/usr/local/lib/python3.9/dist-packages',
 '/usr/lib/python3/dist-packages']

I also have the same experience as yours, and I noticed many of your observations. I could not execute packages in notebook that I installed with mamba. I could use mamba installed packages on terminal, but not in the notebook. So, I just installed packages I needed using pip -U --user command. So far, I have not run into any issue from using pip.

Does this answer your question?

#67

Great, thanks for confirming. I also just found this post from Jeremy:

GitHub - fastai/paperspace-setup: Setup a paperspace instance for fastai

Which seems also to be aimed at the new setup, where mamba comes pre-installed.

1 Like
#68

Hi there,

Thanks so much for these videos, it’s great to spend so much time on the setup and get comfortable with it :pray:

There is one last thing I would like to understand better, the .bashrc and .bash_profile scripts.

The background is that I noticed that when we enter the terminal in jupyter lab, the aliases set up in the .bashrc script are not working (ll, ls etc), as well as the coloring of folders/files when executing ls (which is configured in .bashrc) is not working.

However, when executing bash in the terminal all this stuff seems to work all of a sudden [Also, the conda environment is being activated, as shown by the (base) on the start of each line, which is also part of the .bashrc script]. So somehow entering a new bash ran this .bashrc script.

I read somewhere that the .bashrc script is executed whenever we are entering a “non-login shell” and the .bash_profile script whenever we enter the “login shell”.

I found out here that we can run

shopt -q login_shell && echo 'Login shell' || echo 'Non-login shell'

to find out whether we are in a login or non-login shell.

Running this command in the jupyter terminal gives that we are running in login shell, so that indeed the .bash_profile script should be executed and not the .bashrc

However, I also spin up a totally empty machine (without fastai or anything) on paperspace and found that a very similar .bashrc is created, so with all the coloring and aliases included. However, opening a terminal on that machine seems to run this script since coloring and aliases are working directly. I guess this is because there exist no bash_profile script by default, so instead it executes the bashrc. And indeed upon touch .bash_profile and opening a new terminal the coloring and aliases are gone, because now it is executing the (empty) .bash_profile and no longer the .bashrc for login shell’s.

Question is: can somebody shed some light on this, what’s the reason for having a login and non-login shell? Why does only the non-login shell activate the base conda environment? And last but not least, how does this relate to Jeremy having coloring in the terminal upon opening a terminal from (paperspace) jupyter?