JPEG is a lossy compression. Although you may not be able to see the difference, the network might when applying hundreds of filters. 20% drop in accuracy is still pretty extreme.
Does anybody have first hand experience how float16 and float32 compares? i usually use float16 for space saving reasons, but it may be a good idea to up my game.