Installing pillow-simd and libjpeg-turbo in conda environment

Deep Learning
1

Hi,

First of all, apologies if I’m asking any stupid question. I’m pretty new to conda.

I’ve been trying for a couple of days to successfully install pillow-simd and libjpeg-turbo in a conda environment to reduce the time to load images and doing some data augmentations. At times I’ve been successful but I’m not able to make it reproducible.

I generally create the conda environment using the following command:

conda create --name my_env python=3.7 pandas numpy scipy matplotlib jupyterlab xgboost lightgbm catboost pytorch torchvision torchaudio cudatoolkit=11.0 rapids-blazing=0.17 pytorch-lightning dvc kaggle black mypy dask albumentations -c pytorch -c rapidsai -c nvidia -c conda-forge -c defaults

and then run the following script, as suggested in the fastai documentation.

conda uninstall -y --force pillow pil pillow-simd jpeg libtiff libjpeg-turbo
pip   uninstall -y         pillow pil pillow-simd jpeg libtiff libjpeg-turbo
conda install -yc conda-forge libjpeg-turbo
CFLAGS="${CFLAGS} -mavx2" pip install --upgrade --no-cache-dir --force-reinstall --no-binary :all: --compile pillow-simd==7.0.0.post3
conda install -y jpeg libtiff

I then check if pillow-simd is using libjpeg-turbo by using the following script:

import PIL
from PIL import features

print(f"PIL version: {PIL.__version__}")
if features.check_feature('libjpeg_turbo'):
    print("libjpeg-turbo is on")
else:
    print("libjpeg-turbo is NOT on")

Unfortunately, at times I end up having a system with:

PIL version: 8.1.0
libjpeg-turbo is NOT on

Other times I have pillow=7.0.0 installed, but it doesn’t use libjpeg-turbo. Other times it works.

The problem seems to be that libjpeg-turbo requires pillow=8.1.0. To fix that, I’ve changed the command to install libjpeg-turbo to:

conda install -yc conda-forge libjpeg-turbo=2.0.3 pillow=7.0.0

… but this makes the script to verify if PIL uses libjpeg-turbo fails with this error message:

ImportError: The _imaging extension was built for another version of Pillow or PIL:
Core version: 7.0.0.post3
Pillow version: 7.0.0

Has anyone found a robust way to install pillow-simd and libjpeg-turbo alongside other data science libraries?

2

I slightly changed the script and, so far, it seems to successfully install pillow-simd and libjpeg-turbo. All other libraries seem to work so far.

conda uninstall -y --force pillow pil pillow-simd jpeg libtiff libjpeg-turbo
pip   uninstall -y         pillow pil pillow-simd jpeg libtiff libjpeg-turbo
conda install -yc conda-forge libjpeg-turbo
CC="cc -mavx2" pip install --no-cache-dir -U --force-reinstall pillow-simd==7.0.0.post3
conda install -y jpeg libtiff

I’ll update the thread if I encounter any issues.

3

Actually, the results continue to be not reproducible. It seems that every time I run the script, I end up with a different environment state. Very frustrating! :face_with_symbols_over_mouth: