For those who run their own AI box, or want to

prairieguy · May 8, 2022, 8:34pm

Trouble with jupyter_contrib_nbextensions

I get an error related to jupyter_contrib_nbextensions when launching juypter notebook

For those who have this working, suggestions are much appreciated.

I initialize a clean conda env w/ python=3.9 as follows:

conda install -c conda-forge jupyter_contrib_nbextensions
jupyter contrib nbextension install --user
conda install -c conda-forge jupyter_nbextensions_configurator

EDIT: Solved Silly mistake. Only first two steps required:

conda install -c conda-forge jupyter_contrib_nbextensions                                                                                                                                                      
jupyter contrib nbextension install --user

fmussari · May 8, 2022, 9:42pm

Thank you all.

I was activating the environment before running jupyter, but activating it or not which jupyter returned the same (system python) location.

So yes, jupyter was installed in the system python, and that’s the one that got opened - I think I did this accidental install some time ago and not when installing fastai.

So, even though by the time I saw your message I had already installed fastai again (in its own environment), running
python3 -m pip uninstall -y jupyter jupyter_core jupyter-client jupyter-console jupyterlab_pygments notebook qtconsole nbconvert nbformat jupyterlab-widgets nbclient did the job!

And
$ which jupyter
is returning
/home/fmussari/mambaforge/envs/fastaiv3/bin/jupyter

Now I want to reset even WSL2 to get the cleanest possible install!

Thanks again!

balnazzar · May 8, 2022, 9:51pm

It’s strange, nonetheless. Indeed, if you call jupyter from inside a conda env, its jupyter executable takes priority over other jupyters installed elsewhere.

Anyway, do as Jeremy suggested. If you use the whole conda machinery for fastai only, you have no use for multiple envs.
And that for such an use case one can employ the base env without causing messes is new to me too, and it’s ueful to know (e.g. for containerized stuff, etc…).

fmussari · May 8, 2022, 9:58pm

Given that I read Jeremy’s message after reinstalling all in its own environment, by simply uninstalling jupyter from system python did the trick.

In this moment is working while installed in its own environment.

jeremy · May 8, 2022, 11:06pm

It’s not even just in that case - you only need multiple envs if you actually require isolated environments for some particular reason.

Some people do, which is fine - but I’ve not yet needed it myself after many years of python programming. I’ve got dozens of projects on the go at once!

edit: @balnazzar reminded me below about RAPIDS, a library which has complex dependencies, and also testing bleeding edge versions of things – now I think about it, I have indeed used environments for these situations. So I was incorrect to say I’ve never needed them.

Raymond-Wu · May 8, 2022, 7:21am

Having trouble with Anaconda. It’s using a previous version of Python that I had installed but when I do a python --version it returns a different version

Things I’ve tried:

Creating a new conda & mamba environment specifying python version
brew upgrading python
updating conda with conda update python

jeremy · May 8, 2022, 11:11pm

I’m moving this over to the dedicated thread for running on your own machine - this is an advanced approach so not suitable for the beginner category.

balnazzar · May 8, 2022, 11:47pm

Jeremy, pardon me for asking but… Consider my use case. Not a terribly complex ‘expert’ scenario. It’s just that I need fastai, then I need TF & Keras, and RAPIDS, since my company requires me to do stuff with them occasionally. Then I need some additional tools for computational physics. And finally I need some degree of isolation to experiment with new, bleeding-edge, unstable stuff.
A very common scenario.
If I install all that stuff into the base evironment, I expect to fall into a dependency nightmare, where some packages are superseded by others, or downgraded, or unduly upgraded…
Mind that that’s just my guess, for I’ve never tried to do what I (and you) just described.

jeremy · May 8, 2022, 11:52pm

I think you under-estimate your expertise. That’s definitely a complex expert scenario!

TF and RAPIDS both play notoriously poorly with other libraries, and there isn’t really an option but to keep them in isolated environments.

Thanks for your great question/example.

mike.moloch · May 9, 2022, 12:45am

If anyone from Paperspace reads these forums, thank you for making your images available, they have made my life easier for running fastai/fastbook locally! I see that they are being pushed regularly (sometimes almost daily!)

Thanks!

P.S. Is there a manifest of what’s installed on these containers? (Without downloading and ruffling through metadata)

@joshua-paperspace @dillon

bencoman · May 9, 2022, 2:45am

Just to confirm, that means don’t do…

$ mamba create -n fastaiv3 python=3.9.10

jeremy · May 9, 2022, 3:52am

Correct. The only step you should generally need (assuming you’ve gotten rid of all previous installations of conda-related stuff and any libs you’ve added to system python, and restarted your terminal) is to run this script : fastsetup/setup-conda.sh at master · fastai/fastsetup · GitHub

Then you can install fastai.

bencoman · May 9, 2022, 4:00am

[Edit]
I just saw @Jeremy’s reference to fastsetup/setup-conda.sh at master · fastai/fastsetup · GitHub
I’ll try that first when I’m home this evening.
[/Edit]

I’m preparing to install jupyter locally on WSL2 this evening. I’d like to end up with a tested cheat-sheet that I can publish for others new to the system.

From lots of troubleshooting discussion above, the significant points seem to be:

Strongly recommend starting [without creating a new environment]
Environments really are not a great option for beginners, and make debugging issues […] far harder.
The output of that shows that you’ve accidentally installed jupyter into your system python,
and when you ran jupyter you did so when conda wasn’t activated.

…from which I take to mean, Jupyter should be kept out of the system python and run in a conda environment, but just the default base conda environment, not an additionally created environment.
Is that a reasonable interpretation?

@fmussari, you said “Now I want to reset even WSL2 to get the cleanest possible install!”
Did you end up doing that, and perhaps have a record of steps I can cheat off?

From thsi threads discussion I have pieced together the following steps. Anything obvious I’m missing? (note, I haven’t studied Jupyter docs in detail yet, and I can’t try stuff until this evening - just trying to get ahead of the game.)

10 CMD.EXE> wsl --install    
20 $ sudo apt install mamba
30 $ mamba install -c fastchan fastai
40 $ mamba install -c fastchan nbdev
50 $ conda install -c conda-forge jupyter_contrib_nbextensions                                                                                                                                                      
60 $ jupyter contrib nbextension install --user 
70 In Jupyter create a new notebook containing...
> import torch
> import fastai

One thing I’m not sure of is if/how I should be mixing mamba and conda commands.
I also found two references that do things slightly different:

jeremy · May 9, 2022, 5:19am

You don’t need this, since my script installs mamba for you. But even if you did need it, you couldn’t install with apt, since that’s for linux packages, and mamba is a python package – which you’d install using pip, conda, or mamba.

bencoman:

50 $ conda install -c conda-forge jupyter_contrib_nbextensions

Use mamba install instead of conda install to make this faster.

Use --sys-prefix instead of --user to install when using conda/mamba.

Correct. Don’t install anything into your system python. That’s for your system to use, not for you to use!

jeremy · May 9, 2022, 5:20am

If you’ve got mamba, there shouldn’t be any need to ever use conda. mamba is a faster replacement for conda.

joshua-paperspace · May 9, 2022, 1:11pm

Hi @mike.moloch,

I’m happy to hear that you are finding the Paperspace images useful and easy to use!

As for the Fast.ai container specifically, you can find the GitHub repo here: GitHub - Paperspace/fastai-docker: Fast.AI course complete docker container for Paperspace and Gradient.

A lot of the packages in our container come from a pip install of fastai so it may be helpful to look at the Fast.ai Git repo as well: GitHub - fastai/fastai: The fastai deep learning library

bencoman · May 9, 2022, 4:37pm

Local Jupyter From Scratch

In case its of use to others, here is my log of installing Jupyter to run locally on Windows Subystem for Linux, accessed through a web browse running on Windows Desktop. I’d be glad to hear of suggestions for improvements.

The purpose is to faciliate development of the inferencing app, so CPU is sufficient and complexity of GPU for training is not considered.

Installed fresh Ubuntu 20.04 LTS

C:\> wsl --unregister Ubuntu
C:\> wsl --install -d Ubuntu

$ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=20.04
> DISTRIB_CODENAME=focal
> DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"

# Checked internet connectivity, Google DNS server always alive...
$ ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
> 64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=58.5 ms

# Checked DNS
$ ping google.com
> ping: google.com: Temporary failure in name resolution

# The following two steps only required if ping failed.

# Configured DNS - only needed if previous step failed.
$ sudo sh -c 'echo nameserver 8.8.8.8 > /etc/resolv.conf'
$ ping google.com
> PING google.com (142.250.70.206) 56(84) bytes of data.
> 64 bytes from mel05s01-in-f14.1e100.net (142.250.70.206): icmp_seq=1 ttl=56 time=59.7 ms

# Prevented WSL overwriting DNS settings
$ sudo sh -c 'echo "[network]\ngenerateResolvConf = false" > /etc/wsl.conf'
$ cat /etc/wsl.conf
> [network]
> generateResolvConf = false

Installed fast.ai

# Install fastai
$ wget -O - https://raw.githubusercontent.com/fastai/fastsetup/master/setup-conda.sh | /usr/bin/env bash

# Test Jupyter - its not installed
$ jupyter notebook
> (base) ben@OFI-PC-0004:~$ jupyter notebook
> Command 'jupyter' not found, but can be installed with:
> sudo apt install jupyter-core
# DO NOT INSTALL JUPYTER IN SYSTEM, as suggested above, use conda/mamba instead

Installed Jupyter

in base env rather than separate environment.

$ mamba install -c conda-forge notebook
>Looking for: ['notebook']
>Pinned packages:
>  - python 3.9.*
>Transaction
> Prefix: /home/ben/mambaforge
>
>  Change:
>  - certifi                        2021.10.8  py39hf3d152e_1     installed
>  + certifi                        2021.10.8  py39hf3d152e_2     conda-forge/linux-64     148kB
>
>  Upgrade:
>  - openssl                           1.1.1l  h7f98852_0         installed
>  + openssl                           1.1.1o  h166bdaf_0         conda-forge/linux-64       2MB

$ mamba install -c conda-forge nb_conda_kernels

$ mamba install -c conda-forge jupyter_contrib_nbextensions

$ mamba install -c fastchan fastai nbdev

$ which jupyter
> /home/ben/mambaforge/bin/jupyter

Ran jupyter

$ jupyter notebook
>[I 00:10:50.443 NotebookApp] [nb_conda_kernels] enabled, 1 kernels found
>[I 00:10:50.451 NotebookApp] Writing notebook server cookie secret to /home/ben/.local/share/jupyter/runtime/notebook_cookie_secret
>[I 00:10:50.588 NotebookApp] [jupyter_nbextensions_configurator] enabled 0.4.1
>[I 00:10:50.590 NotebookApp] Serving notebooks from local directory: /home/ben
>[I 00:10:50.590 NotebookApp] Jupyter Notebook 6.4.11 is running
>    Or copy and paste one of these URLs:
        http://localhost:8888/?token=5bb5ea9170b3c4cd4d6d32c3e14bd66378026eab8e907d9f

Pasted that URL into Chrome back in Windows Deskop…

Done. Yay!

[Edit:] If your DNS settings later get overridden, see…
https://forums.fast.ai/t/for-those-who-run-their-own-ai-box-or-want-to/96064/159#solved-retaining-wsl-dns-configuration-1

jeremy · May 9, 2022, 4:44pm

That’s odd - do you know what caused that?

bencoman · May 9, 2022, 5:28pm

No, I can only speculate.
Sorry I didn’t pay exact attention to original nameserver IP. It was was something like 172.x.x.x. I see right now the WSL-IP is 172.31.200.178. My WIN11-IP is 10.1.1.116 using the home wireless router as its DNS=10.1.1.1. So it seems WSL is NATd and expects WIN11 to forward its DNS requests to dns-cache-forwarder on the wireless router. That might be blocked by a Windows Firewall rule, or such Win11 service might not exist.

I haven’t played much with the Hyper-V Virtual Switch, but I guess “Internal” means NAT. Perhaps if WSL was conenct externally direct to the wireless network it would get its DNS configured by DHCP response from the wireless router - but I’m not bothering to test that for now. 8.8.8.8 is such an easy number to remember and hasn’t changed in 20 years, so its simpler for me to update resolv.conf than chase other issues.

There is a long discussion here.
Note this is WSL2 on Win11. Apparently WSL1 is okay.

jeremy · May 9, 2022, 7:52pm

I’ve never seen the problem on WSL2 - I see from the github discussion that quite a few people have seen it, but I’ve set up maybe a dozen WSL installs in a range of environments and never came across that issue.

Just mentioning that so people coming across this thread in the future know they shouldn’t change their DNS settings, unless they happen to hit the same issue you did.