Just to confirm, that means don’t do…
$ mamba create -n fastaiv3 python=3.9.10
Just to confirm, that means don’t do…
$ mamba create -n fastaiv3 python=3.9.10
Correct. The only step you should generally need (assuming you’ve gotten rid of all previous installations of conda-related stuff and any libs you’ve added to system python, and restarted your terminal) is to run this script : fastsetup/setup-conda.sh at master · fastai/fastsetup · GitHub
Then you can install fastai.
[Edit]
I just saw @Jeremy’s reference to fastsetup/setup-conda.sh at master · fastai/fastsetup · GitHub
I’ll try that first when I’m home this evening.
[/Edit]
I’m preparing to install jupyter locally on WSL2 this evening. I’d like to end up with a tested cheat-sheet that I can publish for others new to the system.
From lots of troubleshooting discussion above, the significant points seem to be:
…from which I take to mean, Jupyter should be kept out of the system python and run in a conda environment, but just the default base conda environment, not an additionally created environment.
Is that a reasonable interpretation?
@fmussari, you said “Now I want to reset even WSL2 to get the cleanest possible install!”
Did you end up doing that, and perhaps have a record of steps I can cheat off?
From thsi threads discussion I have pieced together the following steps. Anything obvious I’m missing? (note, I haven’t studied Jupyter docs in detail yet, and I can’t try stuff until this evening - just trying to get ahead of the game.)
10 CMD.EXE> wsl --install
20 $ sudo apt install mamba
30 $ mamba install -c fastchan fastai
40 $ mamba install -c fastchan nbdev
50 $ conda install -c conda-forge jupyter_contrib_nbextensions
60 $ jupyter contrib nbextension install --user
70 In Jupyter create a new notebook containing...
> import torch
> import fastai
One thing I’m not sure of is if/how I should be mixing mamba and conda commands.
I also found two references that do things slightly different:
20 $ sudo apt install mamba
You don’t need this, since my script installs mamba for you. But even if you did need it, you couldn’t install with apt
, since that’s for linux packages, and mamba
is a python package – which you’d install using pip
, conda
, or mamba
.
50 $ conda install -c conda-forge jupyter_contrib_nbextensions
Use mamba install
instead of conda install
to make this faster.
60 $ jupyter contrib nbextension install --user
Use --sys-prefix
instead of --user
to install when using conda/mamba.
Jupyter should be kept out of the system python and run in a conda environment, but just the default base conda environment, not an additionally created environment.
Is that a reasonable interpretation?
Correct. Don’t install anything into your system python. That’s for your system to use, not for you to use!
One thing I’m not sure of is if/how I should be mixing mamba and conda commands.
If you’ve got mamba
, there shouldn’t be any need to ever use conda
. mamba is a faster replacement for conda.
Hi @mike.moloch,
I’m happy to hear that you are finding the Paperspace images useful and easy to use!
As for the Fast.ai container specifically, you can find the GitHub repo here: GitHub - Paperspace/fastai-docker: Fast.AI course complete docker container for Paperspace and Gradient.
A lot of the packages in our container come from a pip install of fastai so it may be helpful to look at the Fast.ai Git repo as well: GitHub - fastai/fastai: The fastai deep learning library
In case its of use to others, here is my log of installing Jupyter to run locally on Windows Subystem for Linux, accessed through a web browse running on Windows Desktop. I’d be glad to hear of suggestions for improvements.
The purpose is to faciliate development of the inferencing app, so CPU is sufficient and complexity of GPU for training is not considered.
C:\> wsl --unregister Ubuntu
C:\> wsl --install -d Ubuntu
$ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=20.04
> DISTRIB_CODENAME=focal
> DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"
# Checked internet connectivity, Google DNS server always alive...
$ ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
> 64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=58.5 ms
# Checked DNS
$ ping google.com
> ping: google.com: Temporary failure in name resolution
# The following two steps only required if ping failed.
# Configured DNS - only needed if previous step failed.
$ sudo sh -c 'echo nameserver 8.8.8.8 > /etc/resolv.conf'
$ ping google.com
> PING google.com (142.250.70.206) 56(84) bytes of data.
> 64 bytes from mel05s01-in-f14.1e100.net (142.250.70.206): icmp_seq=1 ttl=56 time=59.7 ms
# Prevented WSL overwriting DNS settings
$ sudo sh -c 'echo "[network]\ngenerateResolvConf = false" > /etc/wsl.conf'
$ cat /etc/wsl.conf
> [network]
> generateResolvConf = false
# Install fastai
$ wget -O - https://raw.githubusercontent.com/fastai/fastsetup/master/setup-conda.sh | /usr/bin/env bash
# Test Jupyter - its not installed
$ jupyter notebook
> (base) ben@OFI-PC-0004:~$ jupyter notebook
> Command 'jupyter' not found, but can be installed with:
> sudo apt install jupyter-core
# DO NOT INSTALL JUPYTER IN SYSTEM, as suggested above, use conda/mamba instead
in base env rather than separate environment.
$ mamba install -c conda-forge notebook
>Looking for: ['notebook']
>Pinned packages:
> - python 3.9.*
>Transaction
> Prefix: /home/ben/mambaforge
>
> Change:
> - certifi 2021.10.8 py39hf3d152e_1 installed
> + certifi 2021.10.8 py39hf3d152e_2 conda-forge/linux-64 148kB
>
> Upgrade:
> - openssl 1.1.1l h7f98852_0 installed
> + openssl 1.1.1o h166bdaf_0 conda-forge/linux-64 2MB
$ mamba install -c conda-forge nb_conda_kernels
$ mamba install -c conda-forge jupyter_contrib_nbextensions
$ mamba install -c fastchan fastai nbdev
$ which jupyter
> /home/ben/mambaforge/bin/jupyter
$ jupyter notebook
>[I 00:10:50.443 NotebookApp] [nb_conda_kernels] enabled, 1 kernels found
>[I 00:10:50.451 NotebookApp] Writing notebook server cookie secret to /home/ben/.local/share/jupyter/runtime/notebook_cookie_secret
>[I 00:10:50.588 NotebookApp] [jupyter_nbextensions_configurator] enabled 0.4.1
>[I 00:10:50.590 NotebookApp] Serving notebooks from local directory: /home/ben
>[I 00:10:50.590 NotebookApp] Jupyter Notebook 6.4.11 is running
> Or copy and paste one of these URLs:
http://localhost:8888/?token=5bb5ea9170b3c4cd4d6d32c3e14bd66378026eab8e907d9f
Pasted that URL into Chrome back in Windows Deskop…
[Edit:] If your DNS settings later get overridden, see…
https://forums.fast.ai/t/for-those-who-run-their-own-ai-box-or-want-to/96064/159#solved-retaining-wsl-dns-configuration-1
# Checked DNS - not configured
That’s odd - do you know what caused that?
No, I can only speculate.
Sorry I didn’t pay exact attention to original nameserver IP. It was was something like 172.x.x.x. I see right now the WSL-IP is 172.31.200.178. My WIN11-IP is 10.1.1.116 using the home wireless router as its DNS=10.1.1.1. So it seems WSL is NATd and expects WIN11 to forward its DNS requests to dns-cache-forwarder on the wireless router. That might be blocked by a Windows Firewall rule, or such Win11 service might not exist.
I haven’t played much with the Hyper-V Virtual Switch, but I guess “Internal” means NAT. Perhaps if WSL was conenct externally direct to the wireless network it would get its DNS configured by DHCP response from the wireless router - but I’m not bothering to test that for now. 8.8.8.8 is such an easy number to remember and hasn’t changed in 20 years, so its simpler for me to update resolv.conf than chase other issues.
There is a long discussion here.
Note this is WSL2 on Win11. Apparently WSL1 is okay.
Note this is WSL2 on Win11. Apparently WSL1 is okay.
I’ve never seen the problem on WSL2 - I see from the github discussion that quite a few people have seen it, but I’ve set up maybe a dozen WSL installs in a range of environments and never came across that issue.
Just mentioning that so people coming across this thread in the future know they shouldn’t change their DNS settings, unless they happen to hit the same issue you did.
btw, do you know the proportion of Win10 or Win11 on those dozen machines. There may be differences. I’m on Win11, but only since a couple of weeks. Also can you share if your WSL connection type is “Internal” (NAT) or “External” (bridged)?
Here says…
With the introduction of WSL 2 Beta, Microsoft has made changes to the system architecture.
The changes include changing from the default bridged network adapter to a hyper-v virtual network adapter.
No need to reply to the rest of this. Just sharing a few things I gathered up from poking around.
Some are a bit of a stretch.
Far down page here says…
disabling TCP/Offload Engine on all network cards fix the issue
This will be interesting for me to look into… ( I don’t have a .wslconfig file yet )
just add .wslconfig in c:\users\yourusername\.wslconfig
with the following:
[wsl2]
localhostForwarding=true
To flesh out my understanding of the environment…
This one is interesting, I’ll keep my eye on…
I noticed that my
$ ip addr
didn’t quite match my\>ipconfig
might be related to…
And I haven’t fully digested the following yet, but looks like a good reference…
Understanding how DNS works in Hyper-V
Not looking for a response. Just sharing tonights experience since having to document it slows me down so I gain a better understanding, and maybe its useful for someone else…
Opened WSL shell.
$ ping google.com
ping: google.com: Temporary failure in name resolution
$ cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory
Wow! thats a suprise, that the system removed that file.
Restarted Windows LxssManager service to reboot Linux Subsystem.
Opened new WSL shell
resolv.conf still missing. Ahh… maybe need…
/etc/wsl.conf
[network]
- generateResolvConf = false
+ generateResolvConf = true
Rebooted Linux Subsystem again and opened new shell, and it had created the file
$ cat /etc/resolv.conf
nameserver 172.31.192.1
$ ifconfig
eth0: inet 172.31.201.159 netmask 255.255.240.0 broadcast 172.31.207.255
# linux host IP and nameserver IP are on the same subnet (172.31.192.1 - 172.31.207.254)
# so expected to be on same layer 2 virtual switch, no routing.
$ ping 172.31.192.1 #nameserver
33 packets transmitted, 0 received, 100% packet loss, time 33284ms
$ ping 10.1.1.116 #Win-host
10 packets transmitted, 0 received, 100% packet loss, time 9393ms
$ ping 10.1.1.1 #Wireless gateway
64 bytes from 10.1.1.1: icmp_seq=1 ttl=63 time=6.39 ms
From Windows side vEthernet(WSL) IP matches linux host. good.
C:> ipconfig /all
Wireless LAN adapter Wi-Fi:
Description . . . . . . . . . . . : Qualcomm QCA61x4A 802.11ac Wireless Adapter
DHCP Enabled. . . . . . . . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.1.1.116(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.1.1.1
DHCP Server . . . . . . . . . . . : 10.1.1.1
DNS Servers . . . . . . . . . . . : 10.1.1.1
Ethernet adapter vEthernet (Default Switch):
Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter
IPv4 Address. . . . . . . . . . . : 172.29.64.1(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.240.0
Ethernet adapter vEthernet (WSL):
Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #2
IPv4 Address. . . . . . . . . . . : 172.31.192.1(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.240.0
$ ping 172.31.201.159 #WSL eth0
Reply from 172.31.201.159: bytes=32 time<1ms TTL=64
$ ping 172.31.192.1 #WSL nameserver
Reply from 172.31.192.1: bytes=32 time<1ms TTL=128
So Win11-host can nameserver, but WSL can’t.
Win11-host can ping WSL, but WSL can’t ping Win11-host.
WSL can ping wireless-router.
Firewall had been turned off during this.
Looking in the registery, it seems that 172.31.192.1 (the expected namserver)
is associated with an interface that no longer shows up for ipconfig. Maybe it was picked up when WSL was installed.
Now I just realised something about, my machine that might be different to others with working DNS. I already had a Windows 10 Virtual Machine configured before I installed WSL. And actually that was created after I had deleted a previous one. Anyway…
So this will be my final post on this.
To avoid the annoyance of /etc/resolve.conf being overwritten with a bad configuration every restart of LxssManager, I’m doing what several forums suggested, and making the file immutable. First need to unlink it from /mnt/wsl/resolv.conf since can’t make that immutable. I using google servers, rather than a local gateway since that would change between home and work.
$ sudo chattr +i /mnt/wsl/resolv.conf
chattr: Inappropriate ioctl for device while reading flags on /mnt/wsl/resolv.conf
$ sudo rm /etc/resolv.conf
$ sudo sh -c 'echo nameserver 8.8.8.8 > /etc/resolv.conf'
$ sudo chattr +i /etc/resolv.conf
Now after restarting LxssManager…
$ cat /etc/resolv.conf
nameserver 8.8.8.8
$ PING google.com (142.250.70.238) 56(84) bytes of data.
64 bytes from mel05s02-in-f14.1e100.net (142.250.70.238): icmp_seq=1 ttl=56 time=61.1 ms
That is sufficient for now.
Not sure if I imagined it, but I think the initial install of an Ubuntu distro had a checkbox for auto generating the resolv.conf. I’m wondering if some paths in the installation are somehow skipping this. I haven’t seen the problem myself, and like Jeremy have loaded various distros across several different machines. As an aside I was pleasantly surprised to see the latest Ubuntu managed to use my NVIDIA Quadro M2000M in my old ZBook G3 GPU. Only 4GB Memory, and slow (2:30 per epoch on cats v dogs) but it worked.
Ubuntu distro had a checkbox for auto generating the resolv.conf
No checkboxes when I install like… wsl --install -d Ubuntu
No, this was through the Microsoft Store install process I recall. If I can get on a clean box I’ll try and find it again. I just added 20.04 to my existing 22.04 and didn’t see the option.
When first starting my local Jupyter install, it was disconcerting that all extensions were disabled OOTB due to compatability.
I see in Lesson 2 video [8:55] Jeremy has cleared that flag, so I presume its generally safe to do so…
But I’m curious about a bit more context on the state of play of nbextensions stability.
A bit of hunting did turn up a comment (2021-09-01) that…
until now nbextensions compatibility test is only for Jupyter Notebook 4.x and 5.x, about 6.x Compatibility testing has not been implemented yet. If this option is selected, under the version 6.x , all functions will be disabled.
And the issue seems to be outstanding for 2.5 years without much stress on this thread…
I'm using latest version of Anaconda 3 (python 3.7.1), and followed setup steps …
So what are your experiences?
Cheers, ben
p.s. here are my Jupyter versions. Would I be correct to say I’m on Jupyter 7.3.1?
$ jupyter --version
IPython : 8.3.0
ipykernel : 6.13.0
ipywidgets : not installed
jupyter_client : 7.3.1
jupyter_core : 4.9.2
jupyter_server : not installed
jupyterlab : not installed
nbclient : 0.6.2
nbconvert : 6.5.0
nbformat : 5.4.0
notebook : 6.4.11
qtconsole : not installed
traitlets : 5.1.1
p.s.2. Which python should I use for new notebooks?..
I wanted to test the doc() function in my new install.
Having done $ mamba install -c fastai nbdev
I thought I would only neeed to do…
I have reviewed nbdev tutorial | nbdev, but nothing stood out.
I can do doc(function_name) and get results without having to install / import nbdev (maybe in my case it is come through fastbook/fastai imports? not sure. It works OOTB for me (fastai 2.6.3)
That’s true - it does work without nbdev. It just doesn’t look quite as cool