OpenCV images (np.array) to Fastai open_image()

Part 1 (2019)
1

Hi! Is there any convenient way to exchange images between OpenCV (np.array) and Fastai (fastai.vision.image.Image)?

I’m working on an Image Points project where the inference results of the previous image are used to transform the next image with OpenCV. To do inference, I currently need to save the OpenCV image as a temporary .png file and read it with Fastai’s open_image() for inference, and repeat the process for every image. This is very inefficient and I’d love to use open_image() on a np.array. Any change it’s possible?

1 Like
2

You’ll need to call PILs Image.fromarray on the cv image (which is just an array) then pil2tensor it before make it a fastai Image()

2 Likes
3

Thanks a bunch, it works like a charm! For anyone else with the same question, here’s how it’s written out (edited with the later correction):

import cv2
from fastai.vision import *
img_cv2 = cv2.imread('rgb1.png')
img_fastai = Image(pil2tensor(img_cv2, dtype=np.float32).div_(255)
print(type(img_fastai))

<class ‘fastai.vision.image.Image’>

4 Likes
4

Actually, after changing the code in the project it turned out it cannot use the np.uint8 type since the model is trained on float. I get the error:

RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same

Changing the dtype to float32 or float 64 just turns the image into a white mess. With other dtypes I get an error similar to above. Any ideas what to do?

5

pil2tensor(img_cv2,np.float32).div_(255)

2 Likes
6

Hey,

I think one more thing you have to be careful is that OpenCV imports image in BGR format instead of RGB format, so you have to use this additional step to convert imported images in RGB format-
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

This step is especially important if you are using a pre-trained model for transfer learning purpose. Hope this helps.

4 Likes
7

Adding .div_(255) solved it. Also, a very good point to keep an eye out for RGB-BGR confusion. Thanks guys!

1 Like
8

Hi all, in fastai course version2 Jeremy mentioned that opencv is much better than PIL. But now in the source code, I can see there is PIL inside the open_image() function. Can anyone explain this?

9

Hi, I need some help regarding this. I think you can help.
I am using open_image (1st converted pixel array into image and saving it) to test my model which is working fine but memory consuming. Therefore, i converted the array image pixel data directly into fastai.vision.image.Image former using
‘img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img_fastai = Image(pil2tensor(img, dtype=np.uint8))’
So, the output image from open_image and from the img_fastai is completely identical but on using learn.predict(’’) for both showing different result.
Open_image one is correct while using img_fastai is incorrect.
any help is appreciated

10

@Tmaloo You might be missing the the mentioned .div_(255) call.

For fastai version 1

For version 1 you can use the following method to convert a PIL (Python Imaging Library) to a fastai Image:

class ImgUtil:
    @staticmethod
    def pil_to_fai_img(img_pil: PIL.Image) -> fastai.vision.image.Image:
        return fastai.vision.image.Image(pil2tensor(img_pil, dtype=np.float32).div_(255))

You can plug the result directly into a predict: learn.predict(ImgUtil.pil_to_fai_img(img_pil))
and thus avoid saving and loading images with fastai.vision.image.open_image().

However, for performance reasons consider using batches of images instead of single images,
if they are available for your application at inference time.

For fastai version 2

AFAIK version 2 will switch completely to opencv images [EDIT: not correct, see muellerzr’s answer below]. So you shouldn’t work with PIL images anymore. You can load opencv images as mentioned before.

11

No, we still use PIL in v2. Its easier to load other types of data in the new framework (via some customizability), but natively its PIL

1 Like
12

Thanks a lot !

13

Hey, has the pil2tensor been renamed or replaced in Fastai2? I can no longer find it and again having the same original problem of switching from cv2 to Fastai.

3 Likes
15

Hello, @eljas1 the given code work fine when it runs fresh and runs on Jupiter.
When I again run it getting issue of like this

72         img = resize(xray)
73         img = to_rgb(img)
---> 74         img = Image(pil2tensor(img, dtype=np.float32).div_(255))
75 #         plt.figure(figsize = (4, 4))
76 #         plt.imshow(img, cmap="gray")

TypeError: 'module' object is not callable

Please help here
is this issue due to jupyter?