Python package management pain

Part 1 (2017)
1

Having been through a lot of pain working through having the wrong version of packages and getting breaks…

Something I have been wondering about… the software package management ecosystem in Python seems very wild and wooly. For instance, Java has Maven and even Javascript has NPM to insure that the proper dependency versions are in use. (NM the constant pain of preferring Windows on my workstation and Python not having platform independent byte code format like those other languages… yeah I know it’s a rougher road, but somebody’s got to do it :slight_smile:)

Is Python left out in the cold for project dependency versioning specification? It’s been a huge pain point for me, and many others I am sure. If there is such a thing, maybe we should be using it in the lessons? The whole bash script thing seem like quite a hack. (although maybe a necessary one in Python world?)

2

python has virtualenv and requirement files that help you manage 3rd party libs for different projects. I still find it a bit clunky to use but it can remove version pains if you need different versions of the same lib for different projects.

3

You should use conda for package management. For most purposes, it’s better and easier to use than virtualenv.

Conda allows for easy installs for many packages (e.g. Scipy) and easy creation of environments (conda create -n env_name python=3 scipy numpy other_cool_package).

You can freeze your environment settings in a YAML File and recreate easily.

2 Likes
4

Ah, this is great tip. So for any packages supported under conda this would be a great solution. And then I looked at pip and it too has “pip freeze”…

$ env1/bin/pip freeze > requirements.txt
$ env2/bin/pip install -r requirements.txt

So I guess my wish would be for the course content to include these frozen dependencies so we understand exactly what environment we need to create.

5

Yep. Combination of conda YAML file and pip requirements should get you everything (and I think there might be a way to store pip requirements in the conda file too although I’ve never tried).

That said, these libraries are changing quickly. You’re much better off trying to recreate everything with the most recent versions than just running someone else’s code.

6

This resource might help you understand the Python packaging ecosystem a bit better: https://packaging.python.org/

1 Like
7

Just beginning the course myself. Can you post your requirements.txt?

Leon

8

Are you setting up an environment for part 1 or part 2?

If part 1, you can use the following if you’re using Anaconda:

conda create -n ml_py2 python=2.7 anaconda

After the envrionment is created, do a source activate ml_py2 and install these other packages (see https://github.com/fastai/courses/blob/master/setup/install-gpu.sh):

conda install -y bcolz
conda upgrade -y --all
pip install theano
pip install keras==1.2.2

I don’t think there is a requirements.txt file available currently, but if you end up creating one with a working environment for part 1 and/or part 2 of this course, I bet folks would be mightly appreciative. Here’s a link on how to do so: https://stackoverflow.com/questions/35802939/install-only-available-packages-using-conda-install-yes-file-requirements-t

-wg