OK, think I’ve resolved the issues, new versions up on the channel. Now everything seems to work pretty well. Importantly I haven’t seen conda flip out and try to redo everything again. I’ll have a look at the feasibility of adding some tests that script conda install to check it plays nicely with other packages to detect any such issues as it’s maintained.
Didn’t test against your full environment Stas (so would still advise cloning), but tested various operations against fastai (and it’s requirement on pillow was the only issue I saw in your environment). Now with an environment with fastai you should be able to just conda install -c defaults -c thomasbrandon -c conda-forge pillow-accel and have it work. The conda-forge channel is for libjpeg-turbo, which I’ll probably look to move into the pillow-accel channel along with libtiff so I can build them together (currently libtiff still uses the non-turbo libjpeg but does work like that). Defaults is just there so it doesn’t re-install dependencies from conda-forge, it works without that but leads to the issue mentioned in your fastai build notes where packages are reinstalled from conda-forge. The only order that matters is putting default before conda-forge to not have it get conda-forge versions of stuff.
(The reinstalling is because of conda’s default behavious of preferring packages from a higher priority channel regardless of version (you can disable this with conda install --no-channel-priority ... or through conda config/.condarc). With the most recent changes it also seems to now always work regardless of channel order which is nice, that was an issue before.)
Installing the pillow-accel package will just install a metapackage and continue using the base pillow package for everything so should always work. So currently this doesn’t really do anything, but I will add a script in this package to check what optimisations are supported and give the appropriate package to install. If you instead (or then after running the checking script) conda install ... pillow-accel-avx2 (or pillow-accel-sse4) it will replace pillow with the specified optimised pillow-simd. You can also conda install -c fastai -c pytorch -c thomasbrandon -c defaults -c conda-forge fastai pillow-accel (-avx2/-sse4 if you know which) and they will install together (again default not really needed, you’d also only need conda-forge if using -avx2 or -sse4).
The only outstanding issue I’m aware of from a user perspective is it’s a little hard to go back from pillow-accel-avx2(/-sse4) to pillow. No single conda command I found will let you remove one package and redo the requirements for the others to go back to the normal pillow packages. This is needed as pillow-accel will install an empty pillow metapackage to satisfy requirements on pillow. So if you conda remove pillow-accel then you’ll get a warning that the environment will be inconsistent and it will leave behind the dummy pillow package to satisfy requirements but then the real pillow isn’t there. A conda install pillow (or conda upgrade) will fix things up. Plus with my most recent changes I think this will work even if you still have the thomasbrandon channel in .condarc. You can remove it without ever having an inconsistent state by first doing a conda install -c ... pillow-accel-sse3 (which I think I’ll rename as noted below) to get the real pillow package, then conda remove pillow-accel to clean all that up. So it’s not too bad, you now get warnings when it is going into an inconsistent state. Though the warning just lists pillow-accel as well as any package depending on pillow (so fastai and torchvision if you have just fastai installed), so doesn’t exactly tell you the real issue or how to fix it, but at least gives some idea. Can’t see any way to avoid this, but of course can give instructions on the github site, and I think I can also add a conda post-link script so when you remove one of the accelerated packages it can warn you about this and suggest a conda update pillow to hopefully resolve any such issues.
Think I’m now pretty happy with the built packages barring any new issues. Not entirely happy with the complexity of the recipe but not sure there’s a much simpler way. Plus think I can now have it so updating to a new version of upstream only requires changing variables at the top of the main recipe, so minimal maintenance needed. And the complexity is at least generally hidden from the user, they just see a few weirdly named packages. I’ll add notes on the github about these and that they aren’t needed in requirements files which can just add a single line to specify a system appropriate version if using a script to deploy, or the generic package which will prompt users if it’s a public library (and probably advising you still may just want to point users at the docs to upgrade themselves).
I’ll look to update the readme with details on how the build works and would be like it if someone verified that the decisions are at least somewhat comprehensible. Wouldn’t want to have disappeared down the rabbit-hole (or up somewhere else) and ended up with something no one else can, or would want to, understand. The build scripts also still need a bit of clean up due to stuff I added in testing both the packages and Azure Pipelines. Then there’s Windows/Mac builds to have a look at. So various things still to look at but if they work then I’m generally reasonably happy with those packages.
I also need to make the various metapackages noarch. Then I can rename the default pillow-accel-sse3 metapackage to pillow-accel-base as it’s not actually tied to sse3 that’s just what the anaconda default builds use. It should install and pull in an appropriate AMD version of pillow if installed there. That way you could depend on pillow-accel and it’d just pull in pillow and then if appropriate give users a message on install about possible optimised packages and the script to check for that.
Oh, and wasn’t looking to use my personal channel going forward, I will look to move it to an appropriately named channel when it’s ready for more general use. Plus when I have a release pipeline on Azure that will upload there so anyone added to the project can push releases there.
On revisions:
Yeah I thought it might be the nightly, but I was able to do a conda install of it so it’s still around. So yeah, guess just a half-baked feature. Maybe the new partnership between Anaconda and Microsoft will give them more resources. Because it is a nice project but obviously a little under-resourced and in need of some better development practices and extra resources to allow them (not that Microsoft are necessarily the paragon there these days).
Tom.