My docker container runs perfectly most of the time but throws “invalid instruction” errors on some AWS instances. Turns out the problem is bcolz which compiles with different flags depending on the CPU. This means it is not portable in a docker image.
Are there any alternatives that are portable and that still allow randomised access to large datasets?
The main alternative I know of is xarray. I don’t know if it would resolve your problem, or if it has all the features we need.
Bcolz have now put a fix on github that tests environment variables in the setup.py to decide what hardware features to include in the compilation. That provides better control and resolves my immediate issue. However I have since noticed warnings from tensorflow re certain hardware features not being enabled e.g. SSE4. Tensorflow do have a number of alternative binaries that can be used.
The learning point is when you build docker images you have to make trade-offs between portability and supporting new hardware features such as AVX, AVX2, SSE4. You can either build for the worst case hardware or have to choose compatible hardware. On AWS there are 100+ different instance types. You can obtain info on some features but not always e.g. I don’t see any mention of SSE4.
FYI TF 1.6 will compile AVX by default