Fastai v1 install issues thread


#103

Fantastic, thank you, I am training dogs_cats without any issues.

I have noticed that the GPU utilization is significantly lower than with fastai 0.7, averaging only 30% compared to nearly 100% before, is this likely to be because I have reduced the number of workers and therefore the speed at which data is fed to the GPU?


#104

Yup, you’re limited by the CPU now since you can’t multiprocess the data fetching.


#105

And I’m guessing there is no easy fix to the multiprocessing issue on windows?


(Andrea de Luca) #106

Hi Fellas. I was playing with V1’s augmentor, and had a nasty surprise. I then replicated the issue inside an Ipython notebook, just to leave out any jupyter-related issue.

Above you can see a python prompt from my personal machine. fastai.vision.transform is imported without any fuss, BUT the kernel does not see the function apply_tmfs() (Still, it is present in image.py).
Below, you can see a python prompt from a machine at my workplace upon which I installed V1. Same install method. The kernel sees apply_tmfs() without any complaint.

Any suggestion?


(Jeremy Howard (Admin)) #107

In the first one you’ve not spelt it correctly. Use tab-completion to help.


(Thomas) #108

I am starting to move to jupyter-lab (instaed of notebook) and in the enviroment I created to use fastai, jupyter notebook imports fastai without problems, but jupyter-lab does not find the library.
Any idea?


(Jeremy Howard (Admin)) #109

We need a lot more info to help you.

I’ll merge this in to the installation thread.


(Andrea de Luca) #110

Thanks boss. You are right, I didn’t spell it correctly since tab-completion didn’t work, and it didnt work since apply_tfms() is not seen. Look:

If I just import it in another way, the kernel does see apply_tfms()! This is madness…

I even wiped the whole environment, did a conda clean --all to flush any broken package in the cache repository, and reinstalled the whole env from scratch. Same behaviour.

Importing the whole fastai and calling vision.transform.apply_tfms() DOES WORK. It doesn’t like from fastai.vision.transform import *, but just for transform, since everything else imported this way does work as well…


(Thomas) #111

From within the jupyter lab, I try to execute the tabular nb:

[1] !conda list
# packages in environment at /Users/thomascapelle/anaconda3/envs/fastai:
#
# Name                    Version                   Build  Channel
appnope                   0.1.0                    py37_0  
asn1crypto                0.24.0                   py37_0  
backcall                  0.1.0                    py37_0  
blas                      1.0                         mkl  
bleach                    3.0.2                    py37_0  
ca-certificates           2018.03.07                    0  
certifi                   2018.10.15               py37_0  
cffi                      1.11.5           py37h6174b99_1  
chardet                   3.0.4                    py37_1  
cryptography              2.3.1            py37hdbc3d79_0  
cycler                    0.10.0                   py37_0  
cymem                     1.31.2           py37h04f5b5a_0  
cytoolz                   0.9.0.1          py37h1de35cc_1  
dataclasses               0.6                        py_0    fastai
dbus                      1.13.2               h760590f_1  
decorator                 4.3.0                    py37_0  
dill                      0.2.8.2                  py37_0  
entrypoints               0.2.3                    py37_2  
expat                     2.2.6                h0a44026_0  
fastai                    1.0.6                      py_1    fastai
fastprogress              0.1.10                     py_0    fastai
freetype                  2.9.1                hb4e5f40_0  
gettext                   0.19.8.1             h15daf44_3  
glib                      2.56.2               hd9629dc_0  
icu                       58.2                 h4b95b61_1  
idna                      2.7                      py37_0  
intel-openmp              2019.0                      118  
ipykernel                 5.0.0            py37h39e3cac_0  
ipython                   7.0.1            py37h39e3cac_0  
ipython_genutils          0.2.0                    py37_0  
ipywidgets                7.4.2                    py37_0  
jedi                      0.13.1                   py37_0  
jinja2                    2.10                     py37_0  
jpeg                      9b                   he5867d9_2  
jsonschema                2.6.0                    py37_0  
jupyter                   1.0.0                    py37_7  
jupyter_client            5.2.3                    py37_0  
jupyter_console           6.0.0                    py37_0  
jupyter_core              4.4.0                    py37_0  
kiwisolver                1.0.1            py37h0a44026_0  
libcxx                    4.0.1                h579ed51_0  
libcxxabi                 4.0.1                hebd6815_0  
libedit                   3.1.20170329         hb402a30_2  
libffi                    3.2.1                h475c297_4  
libgfortran               3.0.1                h93005f0_2  
libiconv                  1.15                 hdd342a3_7  
libpng                    1.6.35               ha441bb4_0  
libsodium                 1.0.16               h3efe00b_0  
libtiff                   4.0.9                hcb84e12_2  
markupsafe                1.0              py37h1de35cc_1  
matplotlib                3.0.0            py37h54f8f79_0  
mistune                   0.8.4            py37h1de35cc_0  
mkl                       2019.0                      118  
mkl_fft                   1.0.6            py37hb8a8100_0  
mkl_random                1.0.1            py37h5d10147_1  
msgpack-numpy             0.4.4.1                  py37_0  
msgpack-python            0.5.6            py37h04f5b5a_1  
murmurhash                0.28.0           py37h0a44026_0  
nbconvert                 5.3.1                    py37_0  
nbformat                  4.4.0                    py37_0  
ncurses                   6.1                  h0a44026_0  
ninja                     1.8.2            py37h04f5b5a_1  
notebook                  5.7.0                    py37_0  
numpy                     1.15.2           py37h6a91979_1  
numpy-base                1.15.2           py37h8a80b8c_1  
olefile                   0.46                     py37_0  
openssl                   1.0.2p               h1de35cc_0  
pandas                    0.23.4           py37h6440ff4_0  
pandoc                    2.2.3.2                       0  
pandocfilters             1.4.2                    py37_1  
parso                     0.3.1                    py37_0  
pcre                      8.42                 h378b8a2_0  
pexpect                   4.6.0                    py37_0  
pickleshare               0.7.5                    py37_0  
pillow                    5.3.0            py37hb68e598_0  
pip                       10.0.1                   py37_0  
plac                      0.9.6                    py37_0  
preshed                   1.0.1            py37h0a44026_0  
prometheus_client         0.4.2                    py37_0  
prompt_toolkit            2.0.6                    py37_0  
ptyprocess                0.6.0                    py37_0  
pycparser                 2.19                     py37_0  
pygments                  2.2.0                    py37_0  
pyopenssl                 18.0.0                   py37_0  
pyparsing                 2.2.2                    py37_0  
pyqt                      5.9.2            py37h655552a_2  
pysocks                   1.6.8                    py37_0  
python                    3.7.0                hc167b69_0  
python-dateutil           2.7.3                    py37_0  
pytorch-nightly-cpu       1.0.0.dev20181014         py3.7_0    pytorch
pytz                      2018.5                   py37_0  
pyzmq                     17.1.2           py37h1de35cc_0  
qt                        5.9.6                h45cd832_2  
qtconsole                 4.4.1                    py37_0  
readline                  7.0                  h1de35cc_5  
regex                     2018.07.11       py37h1de35cc_0  
requests                  2.19.1                   py37_0  
scipy                     1.1.0            py37h28f7352_1  
send2trash                1.5.0                    py37_0  
setuptools                40.4.3                   py37_0  
simplegeneric             0.8.1                    py37_2  
sip                       4.19.8           py37h0a44026_0  
six                       1.11.0                   py37_1  
spacy                     2.0.12           py37h6440ff4_0  
sqlite                    3.25.2               ha441bb4_0  
termcolor                 1.1.0                    py37_1  
terminado                 0.8.1                    py37_1  
testpath                  0.4.2                    py37_0  
thinc                     6.10.3           py37h6440ff4_0  
tk                        8.6.8                ha441bb4_0  
toolz                     0.9.0                    py37_0  
torchvision-nightly-cpu   0.2.1              pyh19dea27_0    fastai
tornado                   5.1.1            py37h1de35cc_0  
tqdm                      4.26.0           py37h28b3542_0  
traitlets                 4.3.2                    py37_0  
typing                    3.6.4                    py37_0  
ujson                     1.35             py37h1de35cc_0  
urllib3                   1.23                     py37_0  
wcwidth                   0.1.7                    py37_0  
webencodings              0.5.1                    py37_1  
wheel                     0.32.1                   py37_0  
widgetsnbextension        3.4.2                    py37_0  
wrapt                     1.10.11          py37h1de35cc_2  
xz                        5.2.4                h1de35cc_4  
zeromq                    4.2.5                h0a44026_1  
zlib                      1.2.11               hf3cbc9b_2 

[2]from fastai import *          # Quick access to most common functionality
   from fastai.tabular import *  # Quick access to tabular functionality
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-eb17ac0e42e6> in <module>()
----> 1 from fastai import *          # Quick access to most common functionality
      2 from fastai.tabular import *  # Quick access to tabular functionality

ModuleNotFoundError: No module named 'fastai'

If I do exactly the same thing, in the jupyter notebook instead of lab, it works.
I am in a mac, with CPU versions of torch.
What other info I need to provide?


(Jeremy Howard (Admin)) #112

Thanks for the extra info. apply_tfms isn’t defined in that module. Generally it’s easiest to simply from fastai import * and from fastai.vision import * (assuming you are doing a vision application) then you’ll have everything you need. You can then type apply_tfms and jupyter will return <function fastai.vision.image.apply_tfms(... so you can see where it came from. You’ll also find it in the vision.image section of the docs:

https://docs.fast.ai/vision.image.html#apply_tfms


(Andrea de Luca) #113

Thanks for your quick and effective reply.

But allow me one more question: If my personal box shows the expected behaviour, why did the DL box at work behave differently?

Thanks!

Yes I was reading it right now, it is a pleasure to read such a well-written documentation. Job well done!


(Stas Bekman) #114

https://docs-dev.fast.ai/troubleshoot.html#managing-multiple-installations


(Thomas) #115

I checked, and reinstalling jupyter-lab solved the issue. There are many threads about conda enviroments not showing in jupyter,


I hope that when jupyter-lab is officially released this is solved.
You can create a new, empty enviroment in conda, and call jupyter-lab/notebook and it will open, calling the system jupyter. This should not be allowed.


(Stas Bekman) #116

@tcapelle , I documented a summary of that SO thread here:
https://docs-dev.fast.ai/troubleshoot#conda-environments-not-showing-up-in-jupyter-notebook
Since I can’t reproduce this issue, please review and let me know if anything needs to be changed in that new section.

If someone can give me a set of commands to run including creating a new conda environment and indicate at which stage the problem shows up - that will help me to reproduce the problem - and solve it.


#117

sorry if that’s the incorrect thread to reply on… do you know when pytorch1.0 is going to be ready in windows (rather than building myself)?

i have tried a whole day, installing vs2017 and follow the steps in pytorch website but can’t get it compiled…
i need to do a trial and error on installing cmake, msbuild, vs2017, v141…etc but still can’t get it done.


#118

They said at the conference the v1 would really be out around NIPS though you can also ask that question on the pytorch forum :wink:


(Matthew Rosenthal) #119

Just wanted to post that I built a new Ubuntu 18.04 box today with a 1080ti and was able to get it all installed fairly straightforward(only 1 full OS re-install).

I mainly followed this guide - http://blog.jeffhaluska.com/adventures-in-installing-pytorch-in-ubuntu-18-04/

Instead of using ubuntu gui(first 2 steps) to install nvidia drivers I did it through apt install the ppa repo:

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-driver-410

I did conda install for pytorch with this:

conda install pytorch-nightly cuda92 -c pytorch


(Chan Sooi Loong) #121

how install the latest version of fastai?
running conda update fastai -c fastai will only install version 1.0.6

UPDATE: manage to install the latest version after doing these steps:

  1. conda update conda -y
  2. conda install anaconda -y

If you encounter any permission error, do this:
sudo chown -R user /home/user/src
where user is the username, /home/user/src is the folder the where anaconda is installed in Ubuntu


(Scott H Hawley) #122

I’m getting “ImportError: libcuda.so.1: cannot open shared object file: No such file or directory”

As per the Troubleshooting thread, I my driver is
nvidia-396/xenial,now 396.54-0ubuntu0~gpu16.04.1 amd64 [installed]
, and nvidia-smi is working.

$ find /usr/ | grep libcuda.so
/usr/share/man/man7/libcuda.so.7
/usr/local/cuda-9.2/lib64/stubs/libcuda.so
/usr/local/cuda-9.2/doc/man/man7/libcuda.so.7

…these are from the CUDA that I just installed tonight in trying to get FastAI working.

On this Ubuntu 16.04 system, previously I was using CUDA 9.1 with CUDNN7.0 with PyTorch and it worked fine with my GPU. The things that are broken tonight are a result of trying to follow the Fast.ai installation instructions.

I’ve created a fresh conda environment, ran the 3 conda install lines to install pytorch and the fastai packages, but when I run the test line, I get

$ python -c 'import fastai; fastai.show_install(1)'
Traceback (most recent call last):
File "&lt;string&gt;", line 1, in &lt;module&gt;
File "/opt/anaconda/envs/fastai/lib/python3.6/site-packages/fastai/__init__.py", line 1, in &lt;module&gt;
from .basic_train import *
File "/opt/anaconda/envs/fastai/lib/python3.6/site-packages/fastai/basic_train.py", line 2, in &lt;module&gt;
from .torch_core import *
File "/opt/anaconda/envs/fastai/lib/python3.6/site-packages/fastai/torch_core.py", line 2, in &lt;module&gt;
from .imports.torch import *
File "/opt/anaconda/envs/fastai/lib/python3.6/site-packages/fastai/imports/__init__.py", line 2, in &lt;module&gt;
from .torch import *
File "/opt/anaconda/envs/fastai/lib/python3.6/site-packages/fastai/imports/torch.py", line 1, in &lt;module&gt;
import torch, torch.nn.functional as F
File "/opt/anaconda/envs/fastai/lib/python3.6/site-packages/torch/__init__.py", line 84, in &lt;module&gt;
from torch._C import *
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

Would appreciate any help! Thanks.

PS- I do need CUDA installed ‘normally’ on my system, because I also have a project that uses Keras that I need to run on this machine. (…Oh geez, and I removed CUDA 9.1 in doing the Troubleshooting, but my Tensorflow was built from source using 9.1!)


(Stas Bekman) #123

There is an issue with the recent fastai conda packages if you have an outdated anaconda package.

This is caused by an outdated anaconda package, which wants a numpy < 1.15

do

conda install anaconda

which should install anaconda 5.3.0 or higher.

and then it should work.

I changed meta.yaml deps to not ask for numpy>=1.15, but rolled back to >=1.12 so the next release should take care of it.