Performance Improvement Through Faster Software Components

fastai
21

No problem. Thanks for reading the responses and asking good questions so I think through the issues.

Update:
After a fair bit of learning conda build (and a bit of fighting it) I’ve now got a seemingly working build running on Azure Pipelines. Just need to do a few final checks and cleanup then I’ll push a test version to anaconda cloud.
Also had reasonable initial success with a local windows build. It builds and passes the selftest but currently isn’t doing compiler flags (which could uncover errors) and needs to be added to Azure. But hopeful on that.
Will have a go at a Mac build as well at some point and see if that goes as smoothly. That’s a little harder given lack of local test system and inability to logon to the Azure hosted agents I’m using. It was a fair bit of trial and error just to get linux working and I could at least inspect my local system for that (not entirely happy with some changes to run on Azure but had a lot of trouble finding various build outputs, will return to that at some point too). But maybe it’ll just work on Mac, the anaconda recipes seem to.

And responses:

I’ve currently just used azure-pipelines rather than conda-forge (so will publish to anaconda cloud). I’m not sure it would work on conda-forge currently as I’m using some fairly new features of conda build which I’m not sure are supported there. But that might be largely based on older information. Also not entirely sure everything would fit into their templates. Plus I wanted to get some understanding of the underlying CI stuff by doing it myself first at least.
It should still have many of the advantages of conda-forge, automatic building of PRs and being easily forkable with free CI coming along (just sign up/in to Azure and point it at your fork).

Ah right I hadn’t noticed that. Though there are other dependencies that are in the main fastai tag.Would the eventual goal be to remove all non-fastai code from the fastai channel? Does that not complicate installation/maintenance, requiring various channels, given not everything is in the default channel, nor likely ever will be. You could just rely on conda-forge for everything but it seems then you’re at the whim of various upstream maintainers and changes there could potentially break fastai without any local changes. Though from what I’ve seen so far conda-forge packages seem pretty good in spite of the very decentralised nature.
One option I thought might have worked was a fastai-depends channel, this limits the number of channels and allows filtering of upstream changes while clearly delineating them from fastai itself. But that’s obviously really a side issue for this. Fitting into fastai is good but shouldn’t be the only consideration.
I guess from my perspective (as at least temporary maintainer of the packages) the main advantage of having a fastai affiliated channel is the wonderful forum would likely make support a fair bit easier by solving some of the problems people have that aren’t necessarily due to the package itself. Though if it’s pointed to by fastai then just a thread here to point support to should provide all that. Though of course if you did want to include the packages in the (or a) fastai channel I’d happily help with that.

On various upstreaming options:
Yeah, looks like Anaconda don’t have that many resources and aren’t really looking to maintain a whole lot of packages. Which is reasonable but does present some issues for those packaging on it. Certainly I wouldn’t expect to get new packages in there. But they may perhaps accept a PR for their existing pillow package if it didn’t add significant maintenance effort and provided advantages. Much the same seems true of pillow-simd maintainer. There is perhaps more of an opportunity there as currently they don’t do pillow-simd build for Windows so perhaps if I could get that working that would be considered enough of an advantage to accept a PR even if it did introduce some complexity.
But as you say otherwise it could easily be published as it’s own thing.

Yeah, can’t see you’d move away from that. Images need to be loaded and even if possible not sure you’d see too much advantage of moving that to GPU without massive rewriting or advances in GPU code generation.
Though as I was getting at I guess if loading was slowing things down then (assuming your dataset is too large to just keep loaded images in memory) presumably you could have a pre-processing step that loaded the images, performed any non-augmentation transforms, and then saved pickled tensors. Then your training loop could just load the tensors and do augmentation. Though that’s not really a generally applicable thing so for fastai loading is still gonna be a general thing.

Tom.

1 Like
22

Thank you for sharing the updates - I’m glad to hear you have been making a great progress, @TomB

Here is what we have under fastai conda channel: https://anaconda.org/fastai/repo?label=main

  • fastai - no comment
  • fastprogress - a core requirement and is a part of the fastai project
  • thinc, regex, spacy - are a copy from conda-forge since the anaconda channel isn’t bothering updating their version for many months now :frowning:
  • nvidia-ml-py3 - is now a core requirement - by nobody has released a conda package for it, so we have it.
  • hub should probably be removed as it’s not up-to-date and we now have a script that automates the install
  • dataclasses for py36 is needed for keeping fastai noarch single package - otherwise we need to make a separate anaconda release for py36 and py37, so it’s just a hack.

At the moment we rely on anaconda + pytorch channels for all other dependencies.

Given that pillow-simd/pillow+libjpegturbo are enhancements and not requirements, I see no reason why those can’t be in any other channel - it should be easy to give instructions to upgrade to the faster components. Especially, since many other projects could benefit from these faster components, so it’s better to have them separate and that should also make it easier to find more involved maintainers.

And, of course, there is no reason why this other channel can’t be discussed/maintained at this forum.

23

OK I’ve put up a test version now with Azure Pipeline and Github and uploaded it to the thomasbrandon channel on Anaconda. The package is called pillow-accel (see below).
If you conda install -c thomasbrandon -c defaults -c conda-forge pillow-accel you’ll get the default SSE4 variant (though I’ll probably look to change the default to the standard pillow package so it just works). You can then conda install ... pillow-accel-avx2 to get the AVX2 enabled variant. This will replace the existing SSE4 variant. Anaconda will regard this as a downgrade but once you choose a particular variant it won’t go back to the default variant. I believe this should also mean that even if there was a newer version of one of the other variants it won’t try to update to that though I need to test that (this is needed if I add in the default pillow).

I haven’t tested it especially exhaustively but it all seems to work. It passes the quick selftest.py script in pillow(/-simd) which is run on every build. If you run the full set of tests from pilllow then there is one failure for test_strip_ycbcr_jpeg_1x1_sampling. This is because currently it’s using a libtiff built against the non-turbo libjpeg.so.9 (libjpeg-turbo isn’t planning to support the v9 API). So that test uses libtiff, and so libjpeg v9, to decode a JPEG encoded TIFF and then compares it to a JPEG file which is decoded using libjpeg-turbo resulting in a mismatch. Using the compare_similar method which allows some difference succeeds (you can specify an allowed MSE or similar loss). Don’t think this alone is enough reason to ship a libtiff linked against libjpeg-turbo, but probably will look at that just as a way to distribute all the pillow image handling dependencies compiled with optimisations to see if that offers much performance advantage (I wouldn’t expect too much but maybe at least the faster data-shuffling SIMD instructions allow will speed them up even if auto-vectorisation does not provide much as I kinda suspect).

I’d also like to split up the libjpeg-turbo package into multiple outputs (with the main libjpeg-turbo providing all of them as it currently does). Currently while libjpeg.so.{8,9} happily coexist the packages are both distributing man files and some command line utilities which clobber each other. This creates a lot of error messages during the build when it does the test install which I’d like to get rid of, though conda build doesn’t detect these so the build succeeds (oddly a standard conda install doesn’t give these errors, not sure why). I can then hopefully also provide away to have libjpeg-turbo satisfy a jpeg=8 requirement so it can be used to replace it without having to do a conda install --force.
Though I’d look to update the conda-forge package with that so don’t neccessarily need to distribute it to gain those advantages I still wouldn’t mind getting rid of the dependence on the conda-forge channel as that complicates install a bit. Though of course that could also be dealt with by publishing to conda-forge (though as noted not sure that’s possible at the moment given some new conda build features used).

Couple of test build specific things:
The pillow-accel name isn’t final I just didn’t want to call it pillow-simd during testing. Though there are pros and cons to having a different name, beyond clashing with existing packages:

  • If I call the package pillow then it will just slot in over pillow. Though I could also achieve this by calling it pillow-accel (or -simd) and having that package depend on a pillow metapackage (conda package with no files) that would satisfy the pillow dependency for other packages (at the cost of more installed metapackages so a more complicated environment file if you save it).
  • If call it pillow-accel (or -simd) then this package can depend on the original pillow in the default configuration, that way if you just did a conda install -c ... pillow-accel you’d get the original pillow build, without having to rebuild/redistribute that under an alternate name as would be needed if I call the package pillow. In this way a basic pillow-accel install would work everywhere pillow does (and track it’s versions without any need to update pillow-accel), but it could also provide a check to see if an optimised variant was available for the system.

Overall I’m probably leaning towards not calling it pillow though using those tricks to have it work as a drop-in replacement.

Whether to call it pillow-simd rather than pillow-accel probably depends on getting it integrated there. I’d probably prefer not to give it the same name, and if it covers both pillow and pillow-simd then a different name is arguably more appropriate. As I can’t imagine many packages depend on pillow-simd not using that name doesn’t seem like a major issue. Obviously it would fully credit pillow-simd and reference it in all docs.
The 0.5.3.0 version is just temporary (and only on a branch), I will drop the 0 and follow upstream when I’m ready to deal with older packages being around (and so update build numbers).

Tom.

1 Like
24

This is strange, that conda installed py36 variant in my env that is py37, downgrading everything to py36, even though I see you have those packages built for py37. It installed a really really old pillow-4.0 and removed a bunch of packages.

This is very buggy in conda - it happened several times where when it can’t satisfy some dependency with given python version it does an absolutely ridiculous thing by downgrading the whole env to a different python version.

conda install -c thomasbrandon -c defaults -c conda-forge pillow-accel

output: https://pastebin.com/6vzx1beA

So I couldn’t test it.

Will need to clone my env first in the future before trying again, but I recovered quickly by forcing python-3.7.3 install.

I don’t think there are any packages depending on pillow-simd on conda - since that package doesn’t exist.

One other thing to consider is that if you create a new package name, how will this work with pypi? I guess it’ll be similar to pytorch, which uses torch on pypi, and pytorch on conda.

25

Sorry, should have said to install it into a new environment, or yeah a clone. I (obviously) haven’t tested it much with other packages yet. Glad you recovered. If you hadn’t come across it there’s also conda list --revisions which will show environment changes and conda install --revision N which lets you ‘rollback’ (I believe it just does reversed install/removes so won’t help if something plays badly but hopefully would have worked there).
Not quite sure what caused that but would assume some dependent package of the py37 variant that another package didn’t like and the best (or first at least) config they could agree on was a rollback to py36. An unfortunate side effect of running a full SAT solver in a package manager.
Can you do a conda list --explicit > spec-file.txt and send me that so I can test against it and hopefully sort out what’s happening. Or at least have it play nice so it won’t hose your environment next time :wink:
I copied over a recipe that specified particular versions of dependencies, but looking at other pillow recipes I don’t think it is that particular and another recipe (either conda-forge or anaconda) just specified package without versions. So I’ll try that. That is at build time and they’ll still be partially fixed at run but if I’m selecting a non-standard version that should resolve it. Otherwise I’ll have to track down the conflict.

I wan’t yet considering pip support. I’d tend to think the current method of compiling it for the machine is best there given the lack of system package management. You can build a wheel from conda but not sure how that’d work with the various non-python dynamic linkages. I’ll have a look at that but was only really looking at conda support at the moment.

Tom.

1 Like
26

Oh, that’s a great feature! I didn’t know about it - thank you!

Except, it didn’t work:

conda install --revision 58

PackagesNotFoundError: The following packages are missing from the target environment:
  - pytorch::pytorch-nightly==1.1.0.dev20190414=py3.7_cuda10.0.130_cudnn7.4.2_0
  - conda-forge::libarchive==3.3.3=he5c8681_1003

That’s a great revision system, conda!

I pm’ed you the spec as requested, @TomB.

The only reason i mentioned pip is wrt package naming, but as I suggested it should be fine.

27

Ah, OK, the issue is that pillow is clashing with pillow-accel as they want incompatible freetype libraries. I can relax that version which should resolve that (I’m using a different version to even pillow 5.4.0).
Though that does point out I need to do something about pillow vs. pillow-accel. I’ll probably try to add in the stuff to let it include the current pillow release as the default. Then a conda install pillow-accel should nicely remove pillow and install it’s own stub for it so dependent packages keep working. Hopefully won’t take too much fighting to get conda build to do that.
You might have tried to (on a cloned environment) force-removed pillow and installed pillow-accel. If you wanted to test you could just install only pillow-accel into a new environment and then clone the pillow-simd repo and run pytest from the root of it. That’ll at least establish it runs across machines (though given it’s built and tested on Azure I’m fairly happy on that). Or just wait for a new version where I’ll specifically look at going over an existing pillow…

Ha. I haven’t used that feature much but did use it a bit in testing but only simple environments. Not quite sure what happened there. Looks like you might have force installed some things that are confusing it.

Oh, did you mean distributing through pypi or in terms of installing other things through pip into your conda env? I should investigate the later. I think it might work as pip won’t care what the conda package was called as long as the stuff in site-libraries looks like pillow. But will test that.

Tom.

1 Like
28

I will wait then.

That’s a great revision system, conda!

Ha. I haven’t used that feature much but did use it a bit in testing but only simple environments. Not quite sure what happened there. Looks like you might have force installed some things that are confusing it.

no, no forcing. just yet another half-baked conda feature. (1) is a pytorch-nightly build so conda’s revision system probably nuked the original package and it can no longer find it on conda servers. (2) libarchive - I don’t even have an idea what it is, i.e. I haven’t installed it directly. The problem is that conda gives user no way to recover from this failure, i.e. you can’t rollback to everything else minus the problematic packages.

The only reason i mentioned pip is wrt package naming, but as I suggested it should be fine.

Oh, did you mean distributing through pypi or in terms of installing other things through pip into your conda env? I should investigate the later. I think it might work as pip won’t care what the conda package was called as long as the stuff in site-libraries looks like pillow. But will test that.

I think it is so.

29

OK, think I’ve resolved the issues, new versions up on the channel. Now everything seems to work pretty well. Importantly I haven’t seen conda flip out and try to redo everything again. I’ll have a look at the feasibility of adding some tests that script conda install to check it plays nicely with other packages to detect any such issues as it’s maintained.

Didn’t test against your full environment Stas (so would still advise cloning), but tested various operations against fastai (and it’s requirement on pillow was the only issue I saw in your environment). Now with an environment with fastai you should be able to just conda install -c defaults -c thomasbrandon -c conda-forge pillow-accel and have it work. The conda-forge channel is for libjpeg-turbo, which I’ll probably look to move into the pillow-accel channel along with libtiff so I can build them together (currently libtiff still uses the non-turbo libjpeg but does work like that). Defaults is just there so it doesn’t re-install dependencies from conda-forge, it works without that but leads to the issue mentioned in your fastai build notes where packages are reinstalled from conda-forge. The only order that matters is putting default before conda-forge to not have it get conda-forge versions of stuff.
(The reinstalling is because of conda’s default behavious of preferring packages from a higher priority channel regardless of version (you can disable this with conda install --no-channel-priority ... or through conda config/.condarc). With the most recent changes it also seems to now always work regardless of channel order which is nice, that was an issue before.)

Installing the pillow-accel package will just install a metapackage and continue using the base pillow package for everything so should always work. So currently this doesn’t really do anything, but I will add a script in this package to check what optimisations are supported and give the appropriate package to install. If you instead (or then after running the checking script) conda install ... pillow-accel-avx2 (or pillow-accel-sse4) it will replace pillow with the specified optimised pillow-simd. You can also conda install -c fastai -c pytorch -c thomasbrandon -c defaults -c conda-forge fastai pillow-accel (-avx2/-sse4 if you know which) and they will install together (again default not really needed, you’d also only need conda-forge if using -avx2 or -sse4).

The only outstanding issue I’m aware of from a user perspective is it’s a little hard to go back from pillow-accel-avx2(/-sse4) to pillow. No single conda command I found will let you remove one package and redo the requirements for the others to go back to the normal pillow packages. This is needed as pillow-accel will install an empty pillow metapackage to satisfy requirements on pillow. So if you conda remove pillow-accel then you’ll get a warning that the environment will be inconsistent and it will leave behind the dummy pillow package to satisfy requirements but then the real pillow isn’t there. A conda install pillow (or conda upgrade) will fix things up. Plus with my most recent changes I think this will work even if you still have the thomasbrandon channel in .condarc. You can remove it without ever having an inconsistent state by first doing a conda install -c ... pillow-accel-sse3 (which I think I’ll rename as noted below) to get the real pillow package, then conda remove pillow-accel to clean all that up. So it’s not too bad, you now get warnings when it is going into an inconsistent state. Though the warning just lists pillow-accel as well as any package depending on pillow (so fastai and torchvision if you have just fastai installed), so doesn’t exactly tell you the real issue or how to fix it, but at least gives some idea. Can’t see any way to avoid this, but of course can give instructions on the github site, and I think I can also add a conda post-link script so when you remove one of the accelerated packages it can warn you about this and suggest a conda update pillow to hopefully resolve any such issues.

Think I’m now pretty happy with the built packages barring any new issues. Not entirely happy with the complexity of the recipe but not sure there’s a much simpler way. Plus think I can now have it so updating to a new version of upstream only requires changing variables at the top of the main recipe, so minimal maintenance needed. And the complexity is at least generally hidden from the user, they just see a few weirdly named packages. I’ll add notes on the github about these and that they aren’t needed in requirements files which can just add a single line to specify a system appropriate version if using a script to deploy, or the generic package which will prompt users if it’s a public library (and probably advising you still may just want to point users at the docs to upgrade themselves).
I’ll look to update the readme with details on how the build works and would be like it if someone verified that the decisions are at least somewhat comprehensible. Wouldn’t want to have disappeared down the rabbit-hole (or up somewhere else) and ended up with something no one else can, or would want to, understand. The build scripts also still need a bit of clean up due to stuff I added in testing both the packages and Azure Pipelines. Then there’s Windows/Mac builds to have a look at. So various things still to look at but if they work then I’m generally reasonably happy with those packages.
I also need to make the various metapackages noarch. Then I can rename the default pillow-accel-sse3 metapackage to pillow-accel-base as it’s not actually tied to sse3 that’s just what the anaconda default builds use. It should install and pull in an appropriate AMD version of pillow if installed there. That way you could depend on pillow-accel and it’d just pull in pillow and then if appropriate give users a message on install about possible optimised packages and the script to check for that.
Oh, and wasn’t looking to use my personal channel going forward, I will look to move it to an appropriately named channel when it’s ready for more general use. Plus when I have a release pipeline on Azure that will upload there so anyone added to the project can push releases there.

On revisions:
Yeah I thought it might be the nightly, but I was able to do a conda install of it so it’s still around. So yeah, guess just a half-baked feature. Maybe the new partnership between Anaconda and Microsoft will give them more resources. Because it is a nice project but obviously a little under-resourced and in need of some better development practices and extra resources to allow them (not that Microsoft are necessarily the paragon there these days).

Tom.

2 Likes
31

The very basic package didn’t do what you thought it should I think:

conda install -c defaults -c thomasbrandon -c conda-forge pillow-accel
[...]
The following NEW packages will be INSTALLED:

  pillow-accel       thomasbrandon/linux-64::pillow-accel-5.3.0.post1-sse3_0


Proceed ([y]/n)?

Downloading and Extracting Packages
pillow-accel-5.3.0.p | 65 KB     | ####################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

python -m fastai.utils.check_perf
Running performance checks.

*** libjpeg-turbo status
✘ libjpeg-turbo is not on. It's recommended you install libjpeg-turbo to speed up JPEG decoding. See https://docs.fast.ai/performance.html#libjpeg-turbo

*** Pillow-SIMD status
✘ Running Pillow 6.0.0; It's recommended you install Pillow-SIMD to speed up image resizing and other operations. See https://docs.fast.ai/performance.html#pillow-simd

i.e. it doesn’t see the files from your package.

The only outstanding issue I’m aware of from a user perspective is it’s a little hard to go back from pillow-accel-avx2(/-sse4) to pillow. No single conda command I found will let you remove one package and redo the requirements for the others to go back to the normal pillow packages. This is needed as pillow-accel will install an empty pillow metapackage to satisfy requirements on pillow.

Perhaps for now just adding explicit instructions to uninstall pillow too?

Oh, and wasn’t looking to use my personal channel going forward, I will look to move it to an appropriately named channel when it’s ready for more general use.

+1

On revisions:
Yeah I thought it might be the nightly, but I was able to do a conda install of it so it’s still around. So yeah, guess just a half-baked feature. Maybe the new partnership between Anaconda and Microsoft will give them more resources. Because it is a nice project but obviously a little under-resourced and in need of some better development practices and extra resources to allow them (not that Microsoft are necessarily the paragon there these days).

I completely agree! That is if Microsoft sees that python community as an asset to support and grow and not just an asset to milk.

32

That is what I thought it would do but I should have explained more clearly.
If you do a conda install -c thomasbrandon pillow-accel-avx2 (or pillow-accel-sse4 if on a machine without AVX2 support) then it will replace pillow with the optimised pillow-simd library.
Installing pillow-accel has just installed the default ‘SSE3 version’ (as shown in the pillow-accel-5.3.0.post1-sse3_0 build string) I’ll probably change this from SSE3 to base to be a little more clear, SSE3 is the level of optimisation Anaconda uses by default, so not enabling any of the extra pillow-simd stuff (and not using it’s code). That is a package that for the moment just depends on pillow (non-SIMD), so it’s dependency was satisfied by the pillow you had before. I will add to this package a script to check processor capabilities and give a post-install message about the script for those running x64 processors (Anaconda docs strongly advise against running anything but basic tools in the various install hooks so I’ll just print a message rather than run the detection script). It’s also needed to prevent you installing multiple pillow-simd variants at once so can’t be eliminated. It would also.allow you to specify it as a requirement if you wanted without breaking anything, so appropriate users will get a message on install about the possibility of upgrading. If I make that package noarch you should even be able to install it on non-supported platforms like ARM.

You shouldn’t need to do that, Anaconda will automatically remove that when you install the AVX2 or SSE4 variant. Though it will say it’s downgrading it or superseeding it depending on exact channels used as it gets replaced by a dummy pillow package. This dummy package is currently tagged as an alpha release so you get pillow-5.3.0a at the moment. Though actually I now use a feature to make sure it properly replaces pillow and gets replaced by it so will likely remove this special version as it complicates the recipe. But I’ll add something like pillow_accel_dummy to the build string so it is clearly marked.

Oh and that script won’t work when you install the pillow-simd version as pillow-simd still hasn’t updated to 5.4.0 with the patch to add the feature you check. That feature also doesn’t actually detect that the accelerated libjpeg is used, just that pillow was compiled against it. The python script here will properly detect the actual presence of the SIMD accelerated functions provided by libjpeg-turbo in loaded libjpeg libraries. You need to conda install -c conda-forge pyelftools to enable the full check.

Tom.

1 Like
33

That did the trick.

*** Pillow-SIMD status
✔ Running Pillow-SIMD 5.3.0.post1

I guess later we can fine tune that reporting tool to tell the user which of the AVX/SSE instructions are supported? That’s if we can programmatically derive that information from the module.

And also perhaps it’d be useful to devise or use an existing benchmark so that we can check that indeed pillow w/ libjpeg-turbo and/or SIMD are helpful and worth the extra effort to install? one for decoding and another for transform?

34

Detecting the AVX2/SSE4 variants is actually kind of tricky. I use function names specific to libjpeg-turbo there. But I’m not sure pillow-simd changes any function names, just some implementations. You could just read the binary and detect the byte sequences for SSE4/AVX2 instructions. The scanning.is trivial and there are computer readable documents with the instructions you could pull the sequences from (I even saw a bash script for it). You’ll get false positives as you could have those sequences in data not code and you can’t tell the difference on x86. But a threshold should allow pretty reliable detection.
You could also add a patch to the pillow-simd C library but I’d prefer to avoid that and the need to handle merge conflicts against new versions.

Yeah, that’s up there on the list. Also useful to decide if it’s worth maintaining an optimised build of libtiff rather than just using the defaults channel one. And nice to have some numbers on what advantage you get to decide whether it’s worth the trade-offs for your usage. Install is now fairly easy, but obviously more complication still.

1 Like
35

What about including a tiny .c file that just checks gcc flag defines and sets some variable which can then be checked from python? That way there is no need to patch anything other than inject one c and one python file. i.e. this would extend pillow’s build reporting functionality.

This is how pillow now can report whether jpeg-turbo was used at build time, so why not do the same for SIMD, except we have to rely on gcc flags instead of jpeg.h header.

And perhaps pillow-SIMD would want to have that as a part of its core eventually?

36

Yeah, that’s a possibility. Complicated a little as while adding a function to the C library is easy enough, any C files should just be compiled and linked in automatically, I think those functions won’t be accessible through the python interface. That would require adding them into the existing python module/class initialisation in the C extension. So you’d need a less maintainable patch there. A couple of options that come to mind are:

  1. Using CFFI to call the function. This requires getting CFFI working which from a quick look seems to have some tricks. Making this cross-platform might also complicate things.
  2. You could also use the export checking method I currently use to just check for the added function not call it. But that method is linux only, so would have to find a way for windows/mac. So the CFFI way may be easier. Unfortunately the CFFI library doesn’t seem to give you a list of exported names which would allow you to do this check easily.
  3. Or if you add a function that returns a string about compilation details then you could just search the extension file for the string data rather than calling it. Or I think you could not even have the function and just export the string (no C expert but think you can do this), though might depend on the available export macros. May be a little slower than calling (though think CFFI has initial overhead) but this isn’t regularly called so that’s fine. It is a little hacky but easy to do cross-platform. Given this isn’t likely to be heavily used a quick and dirty solution like this may be fine.

Likely won’t get too much time to spend on this for a few days. Current work doesn’t give as much downtime to play with this.

1 Like
37

Hmm, since we control the build process perhaps it’d be much easier to simply dump the compiler flags into a text file then and add it to the package and simply read it when needed? I’m surprised python doesn’t provide a way to do that already.

It looks like python only gives access to its build options, but there is no such mechanism for python extensions. I spent quite some time searching for a way, but it looks like nobody seems to need this information, the only related question I found on SO is this, which confirms that there is no standard way to set this information and each extension needs to figure out its own way.

I checked that the egg-info directory doesn’t contain any build information, but I think it’s because the modules don’t set it. So there should be a way to save this info in that directory probably via setup tools.

I also looked through pkg_resources, but didn’t find anything useful there.

I found this package that extends setuptools.Command and allows you to insert any custom metadata during setup.py run:

setup(
    [...]
    meta={
        'meta1': 'value1',
        'meta2': 'value2',
    },
)

and then it ends up in the egg file:

mypackage.egg-info/meta.json:{"meta1": "value1", "meta2": "value2"}

which can then be easily accessed via:

distrib = pkg_resources.get_distribution('mypackage')
meta = json.loads(distrib.get_metadata('meta.json'))

note, setuptools-meta is not on pypi, so needs to be installed as:

pip install git+https://github.com/noirbizarre/setuptools-meta

but it’s really just a subclass of setuptools.Command, so can be just copied into the setup.py directly.

This is for pypi, I haven’t tested whether conda will pick it up, but I think it should (and it’ll require copying that modules code over, since it’s not on conda either).

But perhaps autogenerating a simple text file and adding it to the package will be much simpler.

Likely won’t get too much time to spend on this for a few days. Current work doesn’t give as much downtime to play with this.

No rush, @TomB.

38

Yeah, I was thinking about just a separate file but then there’s the issue of the package being overwritten but that file remaining as it’s unique to pillow-accel. But packaging it in metadata largely eliminates that. If someone piecemeal copies files around that’s their issue. Package metadata should remain accurate even if they manage to install another packaged version. So that egg modification could be nice, good find. There’s already a custom setuptools command so could be added in there. Looks like you’d just need to add self.distribution.meta[key] = value in it’s run.
I also hadn’t thought to look at what setuptools can do to binaries, I was only thinking about C. Maybe something in setuptools could help. Will check that out too.

1 Like
39

Great, then egg meta-data should do the trick!

Good point about extraneous file being left behind.

40

Moving from the other thread to here:

Not entirely sure but based on your log I’d say the issue is that you are trying to build pillow-simd in a conda environment using your system compiler (I’m guessing, can’t tell from that output but if you haven’t specifically installed a compiler into your environment it will be your system one). It is thus looking for a jpeg package in your system not the environment (where libjpeg-turbo is) and failing. If this is the issue then it should be solved by building with a compiler in your environment. Installing jpeg on your system might solve it but using different compilers can cause issues and then your environment depends on system packages which isn’t ideal. You can install a compiler with conda install gxx_linux-64 pkgconfig (pkgconfig is used to locate installed libraries and so ensures it uses the environment ones not system ones). Then do the pip build step. Also note that you haven’t uninstalled pillow so while pillow-simd should overwrite the pillow files it could end up with a mix of files and break.

Or you could try the pillow-accel package I made. It’s still in development and hasn’t been extensively tested but I’ve run all the pillow tests against it without issue (well, one failure but not a major issue, it works just doesn’t pass the test). To install the AVX2 version run conda install -c defaults -c thomasbrandon -c conda-forge pillow-accel-avx2 (while differently named it is just pillow-simd code, the build can be seen on the Azure Pipelines project linked above and related github).
If installing it be sure to review what package changes conda install wants to make. There can be issues where conflicting dependencies cause conda to do silly things like try and switch everything to a different python version. But I think those issues should be largely resolved. One particular issue is that it currently blocks libjpeg v8 as this is replaced by libjpeg-turbo, so if another installed package wants this particular version it will want to uninstall that other package. Generally other packages will install the newer libjpeg v9 which can coexist with libjpeg-turbo which doesn’t support the v9 API. But this isn’t especially tested yet.

2 Likes
41

Thanks for the answer.

Installing a compiler:

conda install gxx_linux-64 pkgconfig

and then trying to compile pillow-simd:

CC="cc -mavx2" pip install --no-cache-dir -U --force-reinstall --no-binary :all: --compile pillow-simd

This this failed with the same error message.

However, installing the AVX2 version worked:

conda install -c defaults -c thomasbrandon -c conda-forge pillow-accel-avx2