Performance Improvement Through Faster Software Components


(Stas Bekman) #1

This thread is dedicated to tips and tricks on improving performance of your ML/DL code, without making any changes to your code or upgrading your hardware.

First, note a new document: https://docs.fast.ai/performance.html - where we would like to compile all kinds of performance improvements, so please feel free to contribute.

As of this writing, the bulk of it is Faster Image Processing. But this hopefully will change over time as you contribute performance improvements in other domains.

Some of the discussed performance improvement checks can be automated. Here is the very first version of the automated checker. It will become available in fastai-1.0.38, until then use git pull and then run:

python -m fastai.utils.check_perf

Currently on my setup I get:

Running performance checks.

*** libjpeg-turbo status
❓ libjpeg-turbo's status can't be derived - need Pillow(-SIMD)? >= 5.4.0 to tell, current version 5.3.0.post0
5.4.0 is not yet available, other than the dev version on github, which can be installed via pip from git+https://github.com/python-pillow/Pillow. See https://docs.fast.ai/performance.html#libjpeg-turbo

*** Pillow-SIMD status
✔ Running Pillow-SIMD 5.3.0.post0

*** CUDA status
✔ Running the latest CUDA 10.0.130 with NVIDIA driver 410.79

Refer to https://docs.fast.ai/performance.html to make sense out of these checks and suggestions.

See also: GPU Optimizations Central

p.s. It’s a wiki post, so anybody can edit it.


GPU Optimizations Central
Developer chat
Developer chat
#2

Want to report my problem maybe useful for others and maybe something to investigate.

Installing fastai on conda virtualenv get me problems trying to install Pillow-SIMD, and without trying to install Pillow-SIMD all is working.

Trying again to recreate the fastai env and install pillow-simd:

  • If I install pillow-simd without uninstalling pillow using the command

from fastai.vision import *

I got an error due the fact pillow-simd is a 5.3.0.post0 release, vs pillow a 5.4.x

  • Uninstalling pillow and installing again pillow-simd on the same command I got the error
    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-3-c0e76450f370> in <module>
    ----> 1 from fastai.vision import *

    ~/anaconda3/envs/fastai-py37/lib/python3.7/site-packages/fastai/vision/__init__.py in <module>
          2 from ..basics import *
          3 from .learner import *
    ----> 4 from .data import *
          5 from .image import *
          6 from .transform import *

    ~/anaconda3/envs/fastai-py37/lib/python3.7/site-packages/fastai/vision/data.py in <module>
        204 
        205 def verify_image(file:Path, idx:int, delete:bool, max_size:Union[int,Tuple[int,int]]=None, dest:Path=None, n_channels:int=3,
    --> 206                  interp=PIL.Image.BILINEAR, ext:str=None, img_format:str=None, resume:bool=False, **kwargs):
        207     "Check if the image in `file` exists, maybe resize it and copy it in `dest`."
        208     try:

    AttributeError: module 'PIL' has no attribute 'Image'

Reinstalling pillow 5.3.0 got the same error.

Something is going on, no idea what but a pity I can’t use the SIMD accelleration.

Dind’t know if have something to do with my hw using Ryzen CPU and a dual GPU setup (AMD and Nvidia).


Fastai v1 install issues thread
(Stas Bekman) #3

Thank you for sharing this info, @davide445.

Didn’t know if have something to do with my hw using Ryzen CPU and a dual GPU setup (AMD and Nvidia).

Are you using the custom packages I made here? https://docs.fast.ai/performance.html#conda-packages

It’s very possible that there is some kind of platform mismatch, since I built those on intel cpu. Can anybody else who is not on the intel cpu confirm whether the conda packages work for you or not?

I asked about this here https://github.com/uploadcare/pillow-simd/issues/44 but got no reply.

Have you tried building from source? https://github.com/uploadcare/pillow-simd#installation

I got an error due the fact pillow-simd is a 5.3.0.post0 release, vs pillow a 5.4.x

It always helps to paste the exact error message. and the exact command that generated it and any other related commands, plus the output of show_install https://docs.fast.ai/support.html#reporting-installation-issues


#4

Are you using the custom packages I made here? https://docs.fast.ai/performance.html#conda-packages

No, I just followed the steps 1-4. What I might have done wrong is mixing conda and pip installation.
Pardon me as a beginner can you please confirm this are the correct steps, your installaton instructions are not clear to me.

  1. conda uninstall -y --force pillow pil jpeg libtiff
  2. conda install -c conda-forge libjpeg-turbo
  3. conda install -c fastai/label/test pillow-simd
  4. conda install -y jpeg libtiff

(Stas Bekman) #5

Yes, you’re using the custom experimental packages. They are invisible to general public, unless they know to include -c fastai/label/test.

So try building from source:

conda uninstall pillow pillow-simd jpeg libtiff --force
pip   uninstall pillow pillow-simd jpeg libtiff 
conda install -c conda-forge libjpeg-turbo
CC="cc -mavx2" pip install -U --force-reinstall pillow-simd

There is no problem mixing conda and pip in this case. But you can build from source only with pip, unless you want to build your own conda package. You will find notes on how to do that here: https://github.com/fastai/fastai/blob/master/builds/custom-conda-builds/pillow-simd/conda-build.txt (i.e. that’s how those experimental packages were built).

But first try the approach I listed above and see if that works. Most likely those conda packages will only work on a similar CPU.


#6

Just to report my previous sequence does have problems so can’t be the correct one.

conda install -c fastai/label/test pillow-simd

After new installations and upgrade generate this

The following packages will be DOWNGRADED:

certifi:           2018.11.29-py37_1000                 conda-forge       --> 2018.11.29-py36_0                
mkl:               2019.1-144                                             --> 2018.0.3-1                       
mkl_fft:           1.0.10-py37ha843d7b_0                                  --> 1.0.6-py36h7dd41cf_0             
mkl_random:        1.0.2-py37hd81dba3_0                                   --> 1.0.1-py36h4414c95_1             
numexpr:           2.6.9-py37h9e4a6bb_0                                   --> 2.6.8-py36hd89afb7_0             
python:            3.7.1-hd21baee_1000                  conda-forge       --> 3.6.8-h0371630_0                 
pytorch:           1.0.0-py3.7_cuda9.0.176_cudnn7.4.1_1 pytorch           --> 0.4.1-py36ha74772b_0             
scipy:             1.2.0-py37h7c811a0_0                                   --> 1.1.0-py36hfa4b5c9_1             
spacy:             2.0.18-py37hf484d3e_1000             fastai            --> 2.0.16-py36h962f231_0            
thinc:             6.12.1-py37h637b7d7_1000             fastai            --> 6.12.1-py36h4989274_0      

So pytorch will be downgraded from 1.0.0 to 0.4.1, I suppose can’t be correct.

Allowing this and also the 4. step checking the environment result in

(fastai-py37) dz@DSPC:~$ python -m fastai.utils.check_perf
Traceback (most recent call last):
  File "/home/dz/anaconda3/envs/fastai-py37/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/dz/anaconda3/envs/fastai-py37/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/dz/anaconda3/envs/fastai-py37/lib/python3.6/site-packages/fastai/utils/check_perf.py", line 1, in <module>
    from ..script import *
  File "/home/dz/anaconda3/envs/fastai-py37/lib/python3.6/site-packages/fastai/script.py", line 2, in <module>
    from dataclasses import dataclass
ModuleNotFoundError: No module named 'dataclasses'  

So this is not the right route.


(Stas Bekman) #7

Where do you see it suggesting to downgrade to pytorch-0.4.1? The output you pasted shows 1.0.0.

ModuleNotFoundError: No module named 'dataclasses'  

How did you install fastai? dataclasses is a dependency that it installs for py36.

Ah! It looks like conda env was created for py37, so fastai didn’t install dataclasses (built in in py37), but then you’re running py36. Somewhere your conda env got downgraded to py36. That’s the problem. Either move back to py37, or reinstall fastai which will install dataclasses.


#8

Your other method didn’t generate errors, but

(fastai-py37) dz@DSPC:~$ python -m fastai.utils.check_perf
Running performance checks.

*** libjpeg-turbo status
❓ libjpeg-turbo's status can't be derived - need Pillow(-SIMD)? >= 5.4.0 to tell, current version 5.3.0.post0

*** Pillow-SIMD status
✔ Running Pillow-SIMD 5.3.0.post0

*** CUDA status
✘ You are running pytorch built against cuda 9.0.176, your NVIDIA driver 410.79 supports cuda10. See https://pytorch.org/get-started/locally/ to install pytorch built against the faster CUDA version.

Refer to https://docs.fast.ai/performance.html to make sense out of these checks and suggestions.

So maybe something is not correct considering “libjpeg-turbo’s status can’t be derived”?

About the previous path py37 was downgraded to py36 from the pillow-simd installation.


(Stas Bekman) #9

libjpeg-turbo’s status feature was added in pillow-5.4.0, and pillow-simd-5.4.0 that should port that feature hasn’t been released, that’s why I can’t tell you programmatically whether it’s enabled or not. Does it make sense?

You may need to tell the pillow-simd developer to update his fork to sync with the pillow-5.4.0

Yes, I was only able to build the conda package against py36. conda build of pillow-simd w/ py37 had compilation problems. There was a talk about fixing the toolchain for py37, I’m not sure whether it happened.

So until it’s all sorted out, were you able to build pillow-simd from source?


#10

Clear enough. Tried to do what you suggested before,

Ah! It looks like conda env was created for py37, so fastai didn’t install dataclasses (built in in py37), but then you’re running py36. Somewhere your conda env got downgraded to py36. That’s the problem. Either move back to py37, or reinstall fastai which will install dataclasses .

to update again from py36 to py37, but got this error

(fastai-py37_psimd) dz@DSPC:~$ python -m fastai.utils.check_perf
Traceback (most recent call last):
  File "/home/dz/anaconda3/envs/fastai-py37_psimd/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/dz/anaconda3/envs/fastai-py37_psimd/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/dz/anaconda3/envs/fastai-py37_psimd/lib/python3.7/site-packages/fastai/utils/check_perf.py", line 5, in <module>
    call_parse(check_perf)
  File "/home/dz/anaconda3/envs/fastai-py37_psimd/lib/python3.7/site-packages/fastai/script.py", line 40, in call_parse
    func(**args.__dict__)
  File "/home/dz/anaconda3/envs/fastai-py37_psimd/lib/python3.7/site-packages/fastai/utils/collect_env.py", line 153, in check_perf
    from PIL import features, Image
ModuleNotFoundError: No module named 'PIL'

So the path py37 is definitelly not working. Btw updating python back to 3.7.1 I got this message

pytorch: 0.4.1-py36ha74772b_0 --> 0.4.1-py37ha74772b_0

so I can say pythorch was effectively downgraded to 0.4.1 trying to execute

conda install -c fastai/label/test pillow-simd

So the right installation path is for me:

  • conda install -c pytorch -c fastai fastai
  • conda install -c pytorch pytorch cuda100
  • conda uninstall pillow pillow-simd jpeg libtiff --force
  • pip uninstall pillow pillow-simd jpeg libtiff
  • conda install -c conda-forge libjpeg-turbo
  • CC="cc -mavx2" pip install -U --force-reinstall pillow-simd

In your installation instruction is stated

NB: fastai v1 currently supports Linux only, and requires PyTorch v1 and Python 3.6 or later.

You may need to clarify only py36 is supported?


(Stas Bekman) #11

You can’t just go back and force between py36 and py37 without creating a big mess. If you chose to run py37 stick to it. If something is asking you to downgrade to py36, don’t install such packages. This is not so smart on part of conda, which tries to be so strict and proper.

So the path py37 is definitelly not working. Btw updating python back to 3.7.1 I got this message
pytorch: 0.4.1-py36ha74772b_0 --> 0.4.1-py37ha74772b_0

If you look closely, you already had pytorch 0.4.1 installed so it just updated the version of the same package. You need to watch what packages cause such downgrade and not hit Enter without looking.

conda can be amazing in some situations and very problematic to use in some others cases.

I suggest you remove this conda env and start from scratch where you stick to py37 and pytorch-1.0.0. That would be the best course of action based on my experience. don’t use the automatic yes -y and instead manually approve any new packages you install, after watching any suggested upgrade/downgrade notices.

So the right installation path is for me:

  • conda install -c pytorch -c fastai fastai
  • conda install -c pytorch pytorch cuda100
  • conda uninstall pillow pillow-simd jpeg libtiff --force
  • pip uninstall pillow pillow-simd jpeg libtiff
  • conda install -c conda-forge libjpeg-turbo
  • CC=“cc -mavx2” pip install -U --force-reinstall pillow-simd

So this one worked, right?

In your installation instruction is stated

NB: fastai v1 currently supports Linux only, and requires PyTorch v1 and Python 3.6 or later.

You may need to clarify only py36 is supported?

pillow-simd has nothing to do with fastai. This is just a courtesy information/support to the fastai users, who may want to speed things up. fastai uses pillow which works with any py36+. Let me know if this clarifies things, @davide445.

So fastai works requires PyTorch v1 and Python 3.6 or later.

And after this discussion I still don’t know whether the pillow-simd conda packages built on Intel platform will not work on AMD.


#12

Thanks for your explanations, really a beginner here.
Using source code compilation pillow-simd appear to be correctly installed.
I need now just figure out why the new conda virtualenv didn’t show up in jupiter notebook to test it.


#13

I just compile the latest Pillow code for SIMD, now fastai prompts correct status for both libjpeg-turbo and Pillow-SIMD as follow:

python -c "import fastai.utils.collect_env; fastai.utils.collect_env.check_perf()"
Running performance checks.

*** libjpeg-turbo status
✔ libjpeg-turbo is on

*** Pillow-SIMD status
✔ Running Pillow-SIMD 6.0.0.post0

*** CUDA status
❓ Running cpu-only torch version, CUDA check is not relevant

But I’m not sure whether libjpeg-turbo’s status feature is correct added in my compiled Pillow-SIMD.

Here is how I build pillow-simd:

  1. clone both Pillow and Pillow-SIMD
  2. mv Pillow-SIMD/src Pillow-SIMD/src-5.3.0.post0
  3. cp -r Pillow/src Pillow-SIMD/
  4. cp Pillow-SIMD/src-5.3.0.post0/libImaging/Filter*.c Pillow-SIMD/src/libImaging/
  5. cp Pillow-SIMD/src-5.3.0.post0/libImaging/Resample*.c Pillow-SIMD/src/libImaging/
  6. change 6.0.0.dev0 to 6.0.0.post0 in Pillow-SIMD/src/PIL/_version.py
  7. cd Pillow-SIMD
  8. CFLAGS="${CFLAGS} -msse4.2" pip install --upgrade --no-cache-dir --force-reinstall --no-binary :all: --compile .

Note, my CPU is very old(Xeon E5-2670v2) which only has SSE4.2 and no AVX2 instructions. So I use -msse4.2 in my compile command. If your cpu supports AVX2, you can use -mavx2 in your compile command.