This is strange, that conda installed py36 variant in my env that is py37, downgrading everything to py36, even though I see you have those packages built for py37. It installed a really really old pillow-4.0 and removed a bunch of packages.
This is very buggy in conda - it happened several times where when it can’t satisfy some dependency with given python version it does an absolutely ridiculous thing by downgrading the whole env to a different python version.
Will need to clone my env first in the future before trying again, but I recovered quickly by forcing python-3.7.3 install.
I don’t think there are any packages depending on pillow-simd on conda - since that package doesn’t exist.
One other thing to consider is that if you create a new package name, how will this work with pypi? I guess it’ll be similar to pytorch, which uses torch on pypi, and pytorch on conda.
Sorry, should have said to install it into a new environment, or yeah a clone. I (obviously) haven’t tested it much with other packages yet. Glad you recovered. If you hadn’t come across it there’s also conda list --revisions which will show environment changes and conda install --revision N which lets you ‘rollback’ (I believe it just does reversed install/removes so won’t help if something plays badly but hopefully would have worked there).
Not quite sure what caused that but would assume some dependent package of the py37 variant that another package didn’t like and the best (or first at least) config they could agree on was a rollback to py36. An unfortunate side effect of running a full SAT solver in a package manager.
Can you do a conda list --explicit > spec-file.txt and send me that so I can test against it and hopefully sort out what’s happening. Or at least have it play nice so it won’t hose your environment next time
I copied over a recipe that specified particular versions of dependencies, but looking at other pillow recipes I don’t think it is that particular and another recipe (either conda-forge or anaconda) just specified package without versions. So I’ll try that. That is at build time and they’ll still be partially fixed at run but if I’m selecting a non-standard version that should resolve it. Otherwise I’ll have to track down the conflict.
I wan’t yet considering pip support. I’d tend to think the current method of compiling it for the machine is best there given the lack of system package management. You can build a wheel from conda but not sure how that’d work with the various non-python dynamic linkages. I’ll have a look at that but was only really looking at conda support at the moment.
Oh, that’s a great feature! I didn’t know about it - thank you!
Except, it didn’t work:
conda install --revision 58
PackagesNotFoundError: The following packages are missing from the target environment:
- pytorch::pytorch-nightly==1.1.0.dev20190414=py3.7_cuda10.0.130_cudnn7.4.2_0
- conda-forge::libarchive==3.3.3=he5c8681_1003
Ah, OK, the issue is that pillow is clashing with pillow-accel as they want incompatible freetype libraries. I can relax that version which should resolve that (I’m using a different version to even pillow 5.4.0).
Though that does point out I need to do something about pillow vs. pillow-accel. I’ll probably try to add in the stuff to let it include the current pillow release as the default. Then a conda install pillow-accel should nicely remove pillow and install it’s own stub for it so dependent packages keep working. Hopefully won’t take too much fighting to get conda build to do that.
You might have tried to (on a cloned environment) force-removed pillow and installed pillow-accel. If you wanted to test you could just install only pillow-accel into a new environment and then clone the pillow-simd repo and run pytest from the root of it. That’ll at least establish it runs across machines (though given it’s built and tested on Azure I’m fairly happy on that). Or just wait for a new version where I’ll specifically look at going over an existing pillow…
Ha. I haven’t used that feature much but did use it a bit in testing but only simple environments. Not quite sure what happened there. Looks like you might have force installed some things that are confusing it.
Oh, did you mean distributing through pypi or in terms of installing other things through pip into your conda env? I should investigate the later. I think it might work as pip won’t care what the conda package was called as long as the stuff in site-libraries looks like pillow. But will test that.
Ha. I haven’t used that feature much but did use it a bit in testing but only simple environments. Not quite sure what happened there. Looks like you might have force installed some things that are confusing it.
no, no forcing. just yet another half-baked conda feature. (1) is a pytorch-nightly build so conda’s revision system probably nuked the original package and it can no longer find it on conda servers. (2) libarchive - I don’t even have an idea what it is, i.e. I haven’t installed it directly. The problem is that conda gives user no way to recover from this failure, i.e. you can’t rollback to everything else minus the problematic packages.
The only reason i mentioned pip is wrt package naming, but as I suggested it should be fine.
Oh, did you mean distributing through pypi or in terms of installing other things through pip into your conda env? I should investigate the later. I think it might work as pip won’t care what the conda package was called as long as the stuff in site-libraries looks like pillow. But will test that.
OK, think I’ve resolved the issues, new versions up on the channel. Now everything seems to work pretty well. Importantly I haven’t seen conda flip out and try to redo everything again. I’ll have a look at the feasibility of adding some tests that script conda install to check it plays nicely with other packages to detect any such issues as it’s maintained.
Didn’t test against your full environment Stas (so would still advise cloning), but tested various operations against fastai (and it’s requirement on pillow was the only issue I saw in your environment). Now with an environment with fastai you should be able to just conda install -c defaults -c thomasbrandon -c conda-forge pillow-accel and have it work. The conda-forge channel is for libjpeg-turbo, which I’ll probably look to move into the pillow-accel channel along with libtiff so I can build them together (currently libtiff still uses the non-turbo libjpeg but does work like that). Defaults is just there so it doesn’t re-install dependencies from conda-forge, it works without that but leads to the issue mentioned in your fastai build notes where packages are reinstalled from conda-forge. The only order that matters is putting default before conda-forge to not have it get conda-forge versions of stuff.
(The reinstalling is because of conda’s default behavious of preferring packages from a higher priority channel regardless of version (you can disable this with conda install --no-channel-priority ... or through conda config/.condarc). With the most recent changes it also seems to now always work regardless of channel order which is nice, that was an issue before.)
Installing the pillow-accel package will just install a metapackage and continue using the base pillow package for everything so should always work. So currently this doesn’t really do anything, but I will add a script in this package to check what optimisations are supported and give the appropriate package to install. If you instead (or then after running the checking script) conda install ... pillow-accel-avx2 (or pillow-accel-sse4) it will replace pillow with the specified optimised pillow-simd. You can also conda install -c fastai -c pytorch -c thomasbrandon -c defaults -c conda-forge fastai pillow-accel (-avx2/-sse4 if you know which) and they will install together (again default not really needed, you’d also only need conda-forge if using -avx2 or -sse4).
The only outstanding issue I’m aware of from a user perspective is it’s a little hard to go back from pillow-accel-avx2(/-sse4) to pillow. No single conda command I found will let you remove one package and redo the requirements for the others to go back to the normal pillow packages. This is needed as pillow-accel will install an empty pillow metapackage to satisfy requirements on pillow. So if you conda remove pillow-accel then you’ll get a warning that the environment will be inconsistent and it will leave behind the dummy pillow package to satisfy requirements but then the real pillow isn’t there. A conda install pillow (or conda upgrade) will fix things up. Plus with my most recent changes I think this will work even if you still have the thomasbrandon channel in .condarc. You can remove it without ever having an inconsistent state by first doing a conda install -c ... pillow-accel-sse3 (which I think I’ll rename as noted below) to get the real pillow package, then conda remove pillow-accel to clean all that up. So it’s not too bad, you now get warnings when it is going into an inconsistent state. Though the warning just lists pillow-accel as well as any package depending on pillow (so fastai and torchvision if you have just fastai installed), so doesn’t exactly tell you the real issue or how to fix it, but at least gives some idea. Can’t see any way to avoid this, but of course can give instructions on the github site, and I think I can also add a conda post-link script so when you remove one of the accelerated packages it can warn you about this and suggest a conda update pillow to hopefully resolve any such issues.
Think I’m now pretty happy with the built packages barring any new issues. Not entirely happy with the complexity of the recipe but not sure there’s a much simpler way. Plus think I can now have it so updating to a new version of upstream only requires changing variables at the top of the main recipe, so minimal maintenance needed. And the complexity is at least generally hidden from the user, they just see a few weirdly named packages. I’ll add notes on the github about these and that they aren’t needed in requirements files which can just add a single line to specify a system appropriate version if using a script to deploy, or the generic package which will prompt users if it’s a public library (and probably advising you still may just want to point users at the docs to upgrade themselves).
I’ll look to update the readme with details on how the build works and would be like it if someone verified that the decisions are at least somewhat comprehensible. Wouldn’t want to have disappeared down the rabbit-hole (or up somewhere else) and ended up with something no one else can, or would want to, understand. The build scripts also still need a bit of clean up due to stuff I added in testing both the packages and Azure Pipelines. Then there’s Windows/Mac builds to have a look at. So various things still to look at but if they work then I’m generally reasonably happy with those packages.
I also need to make the various metapackages noarch. Then I can rename the default pillow-accel-sse3 metapackage to pillow-accel-base as it’s not actually tied to sse3 that’s just what the anaconda default builds use. It should install and pull in an appropriate AMD version of pillow if installed there. That way you could depend on pillow-accel and it’d just pull in pillow and then if appropriate give users a message on install about possible optimised packages and the script to check for that.
Oh, and wasn’t looking to use my personal channel going forward, I will look to move it to an appropriately named channel when it’s ready for more general use. Plus when I have a release pipeline on Azure that will upload there so anyone added to the project can push releases there.
On revisions:
Yeah I thought it might be the nightly, but I was able to do a conda install of it so it’s still around. So yeah, guess just a half-baked feature. Maybe the new partnership between Anaconda and Microsoft will give them more resources. Because it is a nice project but obviously a little under-resourced and in need of some better development practices and extra resources to allow them (not that Microsoft are necessarily the paragon there these days).
The very basic package didn’t do what you thought it should I think:
conda install -c defaults -c thomasbrandon -c conda-forge pillow-accel
[...]
The following NEW packages will be INSTALLED:
pillow-accel thomasbrandon/linux-64::pillow-accel-5.3.0.post1-sse3_0
Proceed ([y]/n)?
Downloading and Extracting Packages
pillow-accel-5.3.0.p | 65 KB | ####################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
python -m fastai.utils.check_perf
Running performance checks.
*** libjpeg-turbo status
✘ libjpeg-turbo is not on. It's recommended you install libjpeg-turbo to speed up JPEG decoding. See https://docs.fast.ai/performance.html#libjpeg-turbo
*** Pillow-SIMD status
✘ Running Pillow 6.0.0; It's recommended you install Pillow-SIMD to speed up image resizing and other operations. See https://docs.fast.ai/performance.html#pillow-simd
i.e. it doesn’t see the files from your package.
The only outstanding issue I’m aware of from a user perspective is it’s a little hard to go back from pillow-accel-avx2(/-sse4) to pillow. No single conda command I found will let you remove one package and redo the requirements for the others to go back to the normal pillow packages. This is needed as pillow-accel will install an empty pillow metapackage to satisfy requirements on pillow.
Perhaps for now just adding explicit instructions to uninstall pillow too?
Oh, and wasn’t looking to use my personal channel going forward, I will look to move it to an appropriately named channel when it’s ready for more general use.
+1
On revisions:
Yeah I thought it might be the nightly, but I was able to do a conda install of it so it’s still around. So yeah, guess just a half-baked feature. Maybe the new partnership between Anaconda and Microsoft will give them more resources. Because it is a nice project but obviously a little under-resourced and in need of some better development practices and extra resources to allow them (not that Microsoft are necessarily the paragon there these days).
I completely agree! That is if Microsoft sees that python community as an asset to support and grow and not just an asset to milk.
That is what I thought it would do but I should have explained more clearly.
If you do a conda install -c thomasbrandon pillow-accel-avx2 (or pillow-accel-sse4 if on a machine without AVX2 support) then it will replace pillow with the optimised pillow-simd library.
Installing pillow-accel has just installed the default ‘SSE3 version’ (as shown in the pillow-accel-5.3.0.post1-sse3_0 build string) I’ll probably change this from SSE3 to base to be a little more clear, SSE3 is the level of optimisation Anaconda uses by default, so not enabling any of the extra pillow-simd stuff (and not using it’s code). That is a package that for the moment just depends on pillow (non-SIMD), so it’s dependency was satisfied by the pillow you had before. I will add to this package a script to check processor capabilities and give a post-install message about the script for those running x64 processors (Anaconda docs strongly advise against running anything but basic tools in the various install hooks so I’ll just print a message rather than run the detection script). It’s also needed to prevent you installing multiple pillow-simd variants at once so can’t be eliminated. It would also.allow you to specify it as a requirement if you wanted without breaking anything, so appropriate users will get a message on install about the possibility of upgrading. If I make that package noarch you should even be able to install it on non-supported platforms like ARM.
You shouldn’t need to do that, Anaconda will automatically remove that when you install the AVX2 or SSE4 variant. Though it will say it’s downgrading it or superseeding it depending on exact channels used as it gets replaced by a dummy pillow package. This dummy package is currently tagged as an alpha release so you get pillow-5.3.0a at the moment. Though actually I now use a feature to make sure it properly replaces pillow and gets replaced by it so will likely remove this special version as it complicates the recipe. But I’ll add something like pillow_accel_dummy to the build string so it is clearly marked.
Oh and that script won’t work when you install the pillow-simd version as pillow-simd still hasn’t updated to 5.4.0 with the patch to add the feature you check. That feature also doesn’t actually detect that the accelerated libjpeg is used, just that pillow was compiled against it. The python script here will properly detect the actual presence of the SIMD accelerated functions provided by libjpeg-turbo in loaded libjpeg libraries. You need to conda install -c conda-forge pyelftools to enable the full check.
*** Pillow-SIMD status
✔ Running Pillow-SIMD 5.3.0.post1
I guess later we can fine tune that reporting tool to tell the user which of the AVX/SSE instructions are supported? That’s if we can programmatically derive that information from the module.
And also perhaps it’d be useful to devise or use an existing benchmark so that we can check that indeed pillow w/ libjpeg-turbo and/or SIMD are helpful and worth the extra effort to install? one for decoding and another for transform?
Detecting the AVX2/SSE4 variants is actually kind of tricky. I use function names specific to libjpeg-turbo there. But I’m not sure pillow-simd changes any function names, just some implementations. You could just read the binary and detect the byte sequences for SSE4/AVX2 instructions. The scanning.is trivial and there are computer readable documents with the instructions you could pull the sequences from (I even saw a bash script for it). You’ll get false positives as you could have those sequences in data not code and you can’t tell the difference on x86. But a threshold should allow pretty reliable detection.
You could also add a patch to the pillow-simd C library but I’d prefer to avoid that and the need to handle merge conflicts against new versions.
Yeah, that’s up there on the list. Also useful to decide if it’s worth maintaining an optimised build of libtiff rather than just using the defaults channel one. And nice to have some numbers on what advantage you get to decide whether it’s worth the trade-offs for your usage. Install is now fairly easy, but obviously more complication still.
What about including a tiny .c file that just checks gcc flag defines and sets some variable which can then be checked from python? That way there is no need to patch anything other than inject one c and one python file. i.e. this would extend pillow’s build reporting functionality.
This is how pillow now can report whether jpeg-turbo was used at build time, so why not do the same for SIMD, except we have to rely on gcc flags instead of jpeg.h header.
And perhaps pillow-SIMD would want to have that as a part of its core eventually?
Yeah, that’s a possibility. Complicated a little as while adding a function to the C library is easy enough, any C files should just be compiled and linked in automatically, I think those functions won’t be accessible through the python interface. That would require adding them into the existing python module/class initialisation in the C extension. So you’d need a less maintainable patch there. A couple of options that come to mind are:
Using CFFI to call the function. This requires getting CFFI working which from a quick look seems to have some tricks. Making this cross-platform might also complicate things.
You could also use the export checking method I currently use to just check for the added function not call it. But that method is linux only, so would have to find a way for windows/mac. So the CFFI way may be easier. Unfortunately the CFFI library doesn’t seem to give you a list of exported names which would allow you to do this check easily.
Or if you add a function that returns a string about compilation details then you could just search the extension file for the string data rather than calling it. Or I think you could not even have the function and just export the string (no C expert but think you can do this), though might depend on the available export macros. May be a little slower than calling (though think CFFI has initial overhead) but this isn’t regularly called so that’s fine. It is a little hacky but easy to do cross-platform. Given this isn’t likely to be heavily used a quick and dirty solution like this may be fine.
Likely won’t get too much time to spend on this for a few days. Current work doesn’t give as much downtime to play with this.
Hmm, since we control the build process perhaps it’d be much easier to simply dump the compiler flags into a text file then and add it to the package and simply read it when needed? I’m surprised python doesn’t provide a way to do that already.
It looks like python only gives access to its build options, but there is no such mechanism for python extensions. I spent quite some time searching for a way, but it looks like nobody seems to need this information, the only related question I found on SO is this, which confirms that there is no standard way to set this information and each extension needs to figure out its own way.
I checked that the egg-info directory doesn’t contain any build information, but I think it’s because the modules don’t set it. So there should be a way to save this info in that directory probably via setup tools.
I also looked through pkg_resources, but didn’t find anything useful there.
I found this package that extends setuptools.Command and allows you to insert any custom metadata during setup.py run:
This is for pypi, I haven’t tested whether conda will pick it up, but I think it should (and it’ll require copying that modules code over, since it’s not on conda either).
But perhaps autogenerating a simple text file and adding it to the package will be much simpler.
Likely won’t get too much time to spend on this for a few days. Current work doesn’t give as much downtime to play with this.
Yeah, I was thinking about just a separate file but then there’s the issue of the package being overwritten but that file remaining as it’s unique to pillow-accel. But packaging it in metadata largely eliminates that. If someone piecemeal copies files around that’s their issue. Package metadata should remain accurate even if they manage to install another packaged version. So that egg modification could be nice, good find. There’s already a custom setuptools command so could be added in there. Looks like you’d just need to add self.distribution.meta[key] = value in it’s run.
I also hadn’t thought to look at what setuptools can do to binaries, I was only thinking about C. Maybe something in setuptools could help. Will check that out too.
Not entirely sure but based on your log I’d say the issue is that you are trying to build pillow-simd in a conda environment using your system compiler (I’m guessing, can’t tell from that output but if you haven’t specifically installed a compiler into your environment it will be your system one). It is thus looking for a jpeg package in your system not the environment (where libjpeg-turbo is) and failing. If this is the issue then it should be solved by building with a compiler in your environment. Installing jpeg on your system might solve it but using different compilers can cause issues and then your environment depends on system packages which isn’t ideal. You can install a compiler with conda install gxx_linux-64 pkgconfig (pkgconfig is used to locate installed libraries and so ensures it uses the environment ones not system ones). Then do the pip build step. Also note that you haven’t uninstalled pillow so while pillow-simd should overwrite the pillow files it could end up with a mix of files and break.
Or you could try the pillow-accel package I made. It’s still in development and hasn’t been extensively tested but I’ve run all the pillow tests against it without issue (well, one failure but not a major issue, it works just doesn’t pass the test). To install the AVX2 version run conda install -c defaults -c thomasbrandon -c conda-forge pillow-accel-avx2 (while differently named it is just pillow-simd code, the build can be seen on the Azure Pipelines project linked above and related github).
If installing it be sure to review what package changes conda install wants to make. There can be issues where conflicting dependencies cause conda to do silly things like try and switch everything to a different python version. But I think those issues should be largely resolved. One particular issue is that it currently blocks libjpeg v8 as this is replaced by libjpeg-turbo, so if another installed package wants this particular version it will want to uninstall that other package. Generally other packages will install the newer libjpeg v9 which can coexist with libjpeg-turbo which doesn’t support the v9 API. But this isn’t especially tested yet.
This should have no effect if it is the first time you’re installing libjpeg-turbo. I had a previous libjpeg-turbo install, so this addition was necessary. Note you would also need to uninstall libtiff-libjpeg-turbo, if you installed it in the optional step.
Amazing efforts guys.
Do you know if there is an existing Dockerfile which combine Pillow-SIMD + libjpeg-turbo + libtiff ?
For some reason, I struggle much with libtiff support, something with imcompatibilities when trying to conda install with the instructions in the post. @stas did you write the original instructions ?
Do you have any hint ?
Thanks much !
I haven’t built it in a while, but since my 8-ball hasn’t been working too well, it’s impossible to help someone if they don’t provide details of what exactly they did (command lines) and how exactly it failed (output logs). Once you share those I’d be happy to have a look.
I don’t understand about dockerfile, it’s a library you’re building how would a docker be useful for that?
