How to prevent "ValueError: Decompressed Data Too Large"?

Part 1 (2019) UTC+8 China / SE Asia
1

Hi everyone,

First post for me in this community. It seems an exciting place.

I’m using a dataset with ~400k images for multi-label classification. When training the model (and on some occasions when displaying with show_batch) I’m facing “Decompressed Data Too Large” error which is raised from here:

  File "/home/ec2-user/anaconda3/envs/py37/lib/python3.7/site-packages/PIL/PngImagePlugin.py", line 93, in _safe_zlib_decompress
    raise ValueError("Decompressed Data Too Large")

I read here that likely this issue is related to PIL’s decompression bombs prevention mechanism and can be resolved by setting ImageFile.LOAD_TRUNCATED_IMAGES to true. Is there a way to control this flag from within fastai?

Also, I suspect that some of the images might be partially downloaded and that could also be source of my challenge here. Is there a way to increase the verbosity to the level of each image file name or something equivalent, which can help me pinpoint which image may be causing that issue?

Thanks!

1 Like