Conda env update error on Ubuntu 16.04


(Niko Gamulin) #1

Hi,

I followed the Normal Installation guideline:

Normal installation

  1. Download project: git clone https://github.com/fastai/fastai.git
  2. Move into root folder: cd fastai
  3. Set up Python environment: conda env update
  4. Activate Python environment: conda activate fastai
  • If this fails, use instead: source activate fastai

when running conda env update, I got errors, which I pasted below.
I have Ubuntu 16.04 machine withNVIDIA GeForce GTX980 GPU card. conda version is 4.5.4 and pip version is 10.0.1.

I have also tried to activate virtual environment with source activate fastai`, but then, when I ran notebook, I had to install opencv2, graphviz and some other libraries manually and I guess this is not the right approach (haven’t installed all required libraries).

Has anyone managed to successfully install fastai on ubuntu machine?


(Alexandre V. Iolov) #2

I also received an error upon conda env update

Exception:
Traceback (most recent call last):
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py”, line 2869, in _dep_map
return self.__dep_map
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py”, line 2663, in getattr
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py”, line 93, in init
req = REQUIREMENT.parseString(requirement_string)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pyparsing.py”, line 1632, in parseString
raise exc
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pyparsing.py”, line 1622, in parseString
loc, tokens = self._parse( instring, 0 )
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pyparsing.py”, line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pyparsing.py”, line 3395, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pyparsing.py”, line 1383, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pyparsing.py”, line 3183, in parseImpl
raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd (at char 33), (line:1, col:34)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py”, line 2949, in init
super(Requirement, self).init(requirement_string)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py”, line 97, in init
requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at “’; extra '”

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_internal/basecommand.py”, line 141, in main
status = self.run(options, args)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_internal/commands/install.py”, line 299, in run
resolver.resolve(requirement_set)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_internal/resolve.py”, line 102, in resolve
self._resolve_one(requirement_set, req)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_internal/resolve.py”, line 306, in _resolve_one
set(req_to_install.extras) - set(dist.extras)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py”, line 2826, in extras
return [dep for dep in self._dep_map if dep]
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py”, line 2871, in _dep_map
self.__dep_map = self._compute_dependencies()
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py”, line 2881, in _compute_dependencies
reqs.extend(parse_requirements(req))
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py”, line 2942, in parse_requirements
yield Requirement(line)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py”, line 2951, in init
raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at “’; extra '”

CondaValueError: pip returned an error

Afterwards, I get the fastai environment installed (conda env list), but upon running
from fastai.imports import *
I get

ModuleNotFoundError Traceback (most recent call last)
in
1 # This file contains all the main external libs we’ll use
----> 2 from fastai.imports import *

~/tools/fastai/courses/dl1/fastai/imports.py in
1 from IPython.lib.deepreload import reload as dreload
----> 2 import PIL, os, numpy as np, math, collections, threading, json, bcolz, random, scipy, cv2
3 import pandas as pd, pickle, sys, itertools, string, sys, re, datetime, time, shutil, copy
4 import seaborn as sns, matplotlib
5 import IPython, graphviz, sklearn_pandas, sklearn, warnings, pdb

ModuleNotFoundError: No module named ‘cv2’

I am also on Ubuntu 16.04, on the newest conda (4.5.11), pip (9.0.1)


(Ali Toyserkani) #3

Having this exact same issue.


(Brian Smith) #4

Same here. Installing fastai on an Azure Deep Learning VM (Linux) - I haven’t had this issue before (a couple of weeks back it worked fine). Seemed to be while it was trying to install ipywidgets or something right after. Removing ipywidgets from requirements.txt and environments.yml got further before a very similar error - but it still isn’t a working environment. Very odd. I wonder what files changed recently.


#5

Same issue here after trying to conda env update
Using Paperspace machine.


#6

This worked for me.


(Niko Gamulin) #7

I managed to run conda env update without errors after following @ashtonsix post

At first I just removed fastai repository, cloned it and the same error reappeared.

Then I ran
conda env remove -y -n fastai
before
conda env update

After running these two commands, there were no errors (haven’t modified environment.yml file as there have been already changes pushed to fastai repository).

Now the only warnings are the following:

twisted 18.7.0 requires PyHamcrest>=1.9.0, which is not installed.
mkl-random 1.0.1 requires cython, which is not installed.
spacy 2.0.12 has requirement regex==2017.4.5, but you’ll have regex 2018.7.11 which is incompatible.

Afterwards, I have installed mkl-random and twisted with pip:
pip install twisted
pip install mkl-random

Finally I have successfully opened and ran jupyter notebooks


Jupyter problem
(Brian Smith) #8

@jeremy commented out the Jupyter dependencies as per the workaround - so the Git Pull works as you saw. If it isn’t fixed soon (by Jupyter!) then the other installs and dependencies could be added to the environment file too.


(Alexandre V. Iolov) #9

I git pulled and can see that the jupyter-* requirements are commented out in environment.yml

However I still get an error upon running conda env update which then errors upon running from fastai.imports import *

The error seems related to the pyproj package… more specifically it seems to miss the pthread library. Where does conda look for that - in /usr/bin or is conda itself responsible for providing a compliant pthreads lib?

Requirement already satisfied: idna<2.8,>=2.5 in /home/alex/tools/anaconda3/envs/fastai/lib/python3.6/site-packages (from requests->torchtext->-r /home/alex/tools/fastai/condaenv.cvpngguc.requirements.txt (line 5)) (2.7)

Collecting pyproj (from geopandas>=0.3.0->plotnine->-r /home/alex/tools/fastai/condaenv.cvpngguc.requirements.txt (line 9))
Using cached https://files.pythonhosted.org/packages/29/72/5c1888c4948a0c7b736d10e0f0f69966e7c0874a660222ed0a2c2c6daa9f/pyproj-1.9.5.1.tar.gz
Complete output from command python setup.py egg_info:
In file included from src/projects.h:292:0,
from nad2bin.c:7:
src/proj_api.h:147:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
projFileAPI *pj_get_default_fileapi();
^
In file included from nad2bin.c:7:0:
src/projects.h:486:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
void pj_deallocate_grids();
^
nad2bin.c:46:13: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
static void Usage()
^
nad2bin.c: In function ‘main’:
nad2bin.c:362:21: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
!= 4 * ct.lim.lam )
^
In file included from src/projects.h:292:0,
from src/pj_malloc.c:5:
src/proj_api.h:147:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
projFileAPI *pj_get_default_fileapi();
^
In file included from src/pj_malloc.c:5:0:
src/projects.h:486:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
void pj_deallocate_grids();
^
/home/alex/tools/anaconda3/envs/fastai/compiler_compat/ld: cannot find -lpthread
/home/alex/tools/anaconda3/envs/fastai/compiler_compat/ld: cannot find -lc
collect2: error: ld returned 1 exit status
using bundled proj4…
Traceback (most recent call last):
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/distutils/unixccompiler.py”, line 197, in link
self.spawn(linker + ld_args)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/distutils/ccompiler.py”, line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/distutils/spawn.py”, line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File “/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/distutils/spawn.py”, line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command ‘gcc’ failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-xbx4rkj4/pyproj/setup.py", line 74, in <module>
    cc.link_executable(objects, execname, extra_postargs = [ '/MANIFEST' ] if os.name == 'nt' else  None)
  File "/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/distutils/ccompiler.py", line 734, in link_executable
    debug, extra_preargs, extra_postargs, None, target_lang)
  File "/home/alex/tools/anaconda3/envs/fastai/lib/python3.6/distutils/unixccompiler.py", line 199, in link
    raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1

----------------------------------------

Command “python setup.py egg_info” failed with error code 1 in /tmp/pip-install-xbx4rkj4/pyproj/


(Andrew Way) #10

This worked for me! Thx @nikogamulin
I didn’t have any other warnings.


(Pankaj Mathur) #12

The root cause of this sudden up rise in " Error: No module named ‘bcolz’.” but bcolz is already installed " is jupyter_contrib_nbextensions

Look at open issue here =>

Solution:

under fastai repo =>

git pull
yes | jupyter kernelspec remove fastai
conda env remove -y -n fastai

conda env update
source activate fastai
python -m ipykernel install --user --name fastai --display-name "fastai"
source deactivate fastai

jupyter lab &


Google Marketplace VM
(Alexandre V. Iolov) #13

Thanks, I’m not sure we are talking about the same thing. Right now I tried from scratch, env removing and rm -rf ing the fastai folder and then re-cloning fastai.git and runing conva env create and again I get the long error message as pasted above. Reading the message above, generated by conda env create -f environment.yml , more carefully, I am surprised by this line which seems to be the actual error:

" /home/alex/tools/anaconda3/envs/fastai/compiler_compat/ld: cannot find -lpthread
/home/alex/tools/anaconda3/envs/fastai/compiler_compat/ld: cannot find -lc
collect2: error: ld returned 1 exit status
using bundled proj4…"

I read this to mean that ld cannot find the pthreads and the core C libraries. How? Of course, I have the C libraries and libpthread-stubs0-dev. I am running the most vanilla distro I can imagine, Ubuntu 16.04 with a GeForce 960 and an i7.

Is it looking for pthread and c in anaconda distribution somewhere and ignoring the /usr/ versions (would make sense)?

This doesn’t help me:

$ yes | jupyter kernelspec remove fastai
Couldn't find kernel spec(s): fastai