Dealing with 16 bit images

Hi,
I have 16 bit .tiff images and it seems that these are not loaded correctly. (my version is 1.0.38)

I tried sublcassing ImageItemList with overriding the open method with:

def open_image_16bit2rgb( fn ):
    # step 1 : open 16 bit grayscale and convert to int32 and create a view on the image a np.asarray
    a = np.asarray(PIL.Image.open( fn ).convert('I'))
    #step 2: add an 1 dimension so we have height, width, 1 channel
    a = np.expand_dims(a,axis=2)
    #create two extra channels to make it an rgb image
    a = np.repeat(a, 3, axis=2)
    return Image( pil2tensor(a, np.float32 ).div(65535) )

which I found somewhere else, but it didn’t work.
Thanks

Sorry to bother you @sgugger, but I couldn’t find a solution. (I updated to 1.0.42)
So I did this:

class ImageItemList2(ImageItemList):
    def open(self, fn):
        return open_image_16bit2rgb(fn)

I created a databunch and when I do this

db.train_ds[0][0]

everything looks fine. But I get a broken pipe error after I try to load a batch. The dataloader is not created correctly.

  1. How can I fix this?
  2. How can I debug broken pipe errors?

BrokenPipeError Traceback (most recent call last)
in
1 dl = db.train_dl
2 dl2 = dl.dl
----> 3 next(iter(dl2))

\miniconda3\envs\f1\lib\site-packages\torch\utils\data\dataloader.py in iter(self)
817
818 def iter(self):
–> 819 return _DataLoaderIter(self)
820
821 def len(self):

\miniconda3\envs\f1\lib\site-packages\torch\utils\data\dataloader.py in init(self, loader)
558 # before it starts, and del tries to join but will get:
559 # AssertionError: can only join a started process.
–> 560 w.start()
561 self.index_queues.append(index_queue)
562 self.workers.append(w)

\miniconda3\envs\f1\lib\multiprocessing\process.py in start(self)
103 ‘daemonic processes are not allowed to have children’
104 _cleanup()
–> 105 self._popen = self._Popen(self)
106 self._sentinel = self._popen.sentinel
107 # Avoid a refcycle if the target function holds an indirect

\miniconda3\envs\f1\lib\multiprocessing\context.py in _Popen(process_obj)
221 @staticmethod
222 def _Popen(process_obj):
–> 223 return _default_context.get_context().Process._Popen(process_obj)
224
225 class DefaultContext(BaseContext):

\miniconda3\envs\f1\lib\multiprocessing\context.py in _Popen(process_obj)
320 def _Popen(process_obj):
321 from .popen_spawn_win32 import Popen
–> 322 return Popen(process_obj)
323
324 class SpawnContext(BaseContext):

\miniconda3\envs\f1\lib\multiprocessing\popen_spawn_win32.py in init(self, process_obj)
63 try:
64 reduction.dump(prep_data, to_child)
—> 65 reduction.dump(process_obj, to_child)
66 finally:
67 set_spawning_popen(None)

\miniconda3\envs\f1\lib\multiprocessing\reduction.py in dump(obj, file, protocol)
58 def dump(obj, file, protocol=None):
59 ‘’‘Replacement for pickle.dump() using ForkingPickler.’’’
—> 60 ForkingPickler(file, protocol).dump(obj)
61
62 #

BrokenPipeError: [Errno 32] Broken pipe

Broken Pipe Errors are often completely unrelated to fastai, and more on the pytoch side, depending on your software. You can get rid of them by setting num_workers=0 in your databunch but it’s going to be slower.

3 Likes

Hi! In my case I want to load 16 bits png images for chest x-ray images classification. I’ve checked and Pytorch does normalize in the range [0, 1] all images (8 and 16 bits ones). When calling dls.show_batch() all images looks white, which I think is due to the function is not normalizing correctly the values from the 16 bit images [0, 65535]. I don’t care that much about show_batch() but if the model is getting the images as I want to (16 bits [0, 65535] but normalize in the range [0, 1]).