Loading RAW image files into ImageBlock

Part 1 (2020)
21

I don’t think you’ll have to change much.

PILBase does some more stuff than just loading the image, so it would be better to subclass it instead of Image.

The problem might be within your load_raw_image function. Maybe this will help?


def load_raw_image(fn):
    fn=str(fn)
    raw_file=rawpy.imread(fn)
    raw_converted_file=raw_file.raw_image_visible.astype(np.float32)
    ndarr -= ndarr.min() #  make min  0
    ndarr /= ndarr.max() # make max 1

    # if you want to return a unit8 array   
    # scale to 0 and 255
    # ndarr *= 255 
    # convert to uint8
    # ndarr = ndarr.astype(np.uint8) 
    return ndarr


class PILRaw(PILBase): 
    _bypass_type=Image.Image
    _show_args = {'cmap':'viridis'}
    _open_args = {'mode': 'F'} #  F (32-bit floating point pixels)

    @classmethod 
    def create(cls,fn:(Path,str), *args, **kwargs)->None:
        return cls(load_raw_image(fn, *args, **kwargs))
22

Thanks for the ideas!
It really didn’t work for me, because what I really want is at least unit16 type of array.
I believe that a wider range of values (from 0 to 65536, rather than 0 to 255) will make the training much much more efficient. Problem with PILBase is, that it inherits Image.Image, which always stores the values in uint8, no matter how detailed the file it gets (16bits depth, 24btis depth or even 32bits depth). It will lose a lot of data when converting to uint8. It sucks

23

To make it slightly more clarified:
The Image.Image object does open and read files such TIFF, BMP which have 32bit or 24bits depth of bits. But then it converts the values to 0 to 255, of a unit8 type. (8bits depth). So even if we read a file that has a depth of 32bits, and we scale it into 0 to 1 values (like you did). Those values will be just converted to values in between 0 to 255, to be stored in the Image.Image object. Only then fastai will convert this object into a tensor. A lot of loss while so

24

Adjusting the mode in RawBase doesn’t help? It seems you still use 8bit RBG.

25

Good note.
I tried to read the raw file with another mode. But here comes the tricky part:
The creators of Image.Image have limited the modes (“RGB”, “F”, etc) in which you can read an Image.Image object.

You can see here: (link)

_fromarray_typemap = {
    # (shape, typestr) => mode, rawmode
    # first two members of shape are set to one
    ((1, 1), "|b1"): ("1", "1;8"),
    ((1, 1), "|u1"): ("L", "L"),
    ((1, 1), "|i1"): ("I", "I;8"),
    ((1, 1), "<u2"): ("I", "I;16"),
    ((1, 1), ">u2"): ("I", "I;16B"),
    ((1, 1), "<i2"): ("I", "I;16S"),
    ((1, 1), ">i2"): ("I", "I;16BS"),
    ((1, 1), "<u4"): ("I", "I;32"),
    ((1, 1), ">u4"): ("I", "I;32B"),
    ((1, 1), "<i4"): ("I", "I;32S"),
    ((1, 1), ">i4"): ("I", "I;32BS"),
    ((1, 1), "<f4"): ("F", "F;32F"),
    ((1, 1), ">f4"): ("F", "F;32BF"),
    ((1, 1), "<f8"): ("F", "F;64F"),
    ((1, 1), ">f8"): ("F", "F;64BF"),
    ((1, 1, 2), "|u1"): ("LA", "LA"),
    ((1, 1, 3), "|u1"): ("RGB", "RGB"),
    ((1, 1, 4), "|u1"): ("RGBA", "RGBA"),
}

Once you open a file in RGB (shape(h=1,w=1,c=3)), the Image.Image object always afterwards converts the file values to “|u1”, which is probably of unsigned 1 byte depth.
I tried to add another mode of ((1, 1, 3), “<u2”): (“RGB”, “RGB”), to support 16bits or even 32bits ("<u4") but then Image.Image returns an error that says something alike: It’s not supported by Image.Image.

I don’t know how to adjust this any longer from here (can I just remove that raise error line?).

26

Plus, you can see the mode “F” only supports (1,1) images, which means that they have only one channel (probably BW/Gray photos), but not 3 channels (like RGB).

27

Other formats like “BGR;16” also don’t work?
If everything fails, you could try using “F” and split the channels to multiple inputs, then use a callback to concat the inputs before passing them to the model.

28

Interesting ideas.
Btw, what’s that “callback”?

29

Hey man,
Any idea what could hold this function “show_batch” from showing the pair of (input,target) tensors together?

image
image1873×761 103 KB

30

Callbacks are a good way to inject some custom functions into the model training without breaking everything in the training loop. E.g., you could use the before_batch function in a callback to concat your inputs.

Here are the docs: Callbacks | fastai

1 Like
31

show_batch calls the show method of each sample in the batch. Maybe your TensorRawpy doesn’t have one?
show is a function of PILBase

see: fastai/core.py at c47022bf0f8e106868d8b422801ecf18e9903d89 · fastai/fastai · GitHub

32

Thanks! TensorRawpy does have a show method, although show_batch doesn’t seem to call it (as seen in the photo that I added from the right).

Edit: I just checked out your link, where it shows a different show_batch function. So I will keep researching from that point! Maybe I’ll find a way to fix it from here.

Original continued message: (Here I refer to show_batch from source code that is shown in the screenshot that I had added, from the right side of it)

show_batch gets as an argument a bool variable show, so it knows whether call _pre_show_batch method or not.

The weird part about show_batch is that it calls itself (recursion) at some point, and I didn’t expect this…

The one_batch() method seems to work though…

33

Thank you!
After trying to debug here and there, I’ve finally found out that there are some different overdriven prototypes of the show_batch function. I found this and made some changes into:

# Cell
@typedispatch
def show_batch(x:TensorImage, y:TensorImage, samples, ctxs=None, max_n=10, nrows=None, ncols=None, figsize=None, **kwargs):
    if ctxs is None: ctxs = get_grid(min(len(samples), max_n), nrows=nrows, ncols=ncols, add_vert=1, figsize=figsize, double=True)
    for i in range(2):
        ctxs[i::2] = [b.show(ctx=c, **kwargs) for b,c,_ in zip(samples.itemgot(i),ctxs[i::2],range(max_n))]
    return
1 Like