Errors When training the bear image classification model

[owura_coder (Richard Agyemang)]

I get this error when i train to fit the model

/opt/conda/envs/fastai/lib/python3.8/site-packages/PIL/Image.py:951: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images

[mrfabulous1]

Hi owura_coder hope all is well and you are having a good day!
The following posts should help resolve the issue.

Lesson 3 - Official Topic

Hi Chikwado I hope you are having a fun day! Below is a link that may help you, solve your problem. You could use the above to covert all the problem images. You could delete the images causing the problem and replace them with others. Doing a google search shows some possible reasons but many of the posts look quite old. Cheers mrfabulous1

You could also try deleting the images using image cleaner then retrain.
ImageCleaner(ds, idxs, path)

Cheers mrfabulous1

[owura_coder (Richard Agyemang)]

Thanks a lot for your reply. I have implemented that and it’s working perfectly.

[dalden (Dorian Alden)]

@mrfabulous1

ImageCleaner(ds, idxs, path)

Hi, I am also getting the following error
" /opt/conda/envs/fastai/lib/python3.8/site-packages/PIL/Image.py:951: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
warnings.warn(…(many lines of code))."

I am not sure how to tell which image this error is pointing towards. Is there a way to identify the issue image, or to fix all potential trouble images?

I am fairly new to both python and ML.

I was not able to implement and fix as owura_coder so a further nodge in the right direction will be appreciated.

Thanks

To add some more context, I have 4 bear categories within a bear folder where 409 images are saved). The error occurs when I try to run the learner
“learn = cnn_learner(dls, resnet18, metrics=error_rate)
learn.fine_tune(4)”

[jimmiemunyi]

Hello @mrfabulous1. I have checked out your solution and it worked well. However, do you know the real reason of this?

When I was working on my dataset, I googled some stuff suggesting PNG images were causing the issue, and true enough, when I wrote a function to delete the PNG images, the error went away automatically.

So, is it PNG images causing the issue?

[mrfabulous1]

Hi jimmiemunyi hope you are having a wonderful day!
It is not the PNG image that is a problem, it is the fact that there are various ways a PNG image can be formatted, most machine learning libraries expect images in particular formats. For example many use images formatted as numpy arrays. See this tutorial for more information. https://www.tensorflow.org/tutorials/images/classification. Also see this link for a simple explanation of rgba. if you were to convert the images to the rgba format they would probably work fine. This link Example: Draw Partial Opacity Text shows an example of converting an rgb image to a rgba image, it adds some transparency which is what the a in rgba configures.

Hope this helps.
Cheers mrfabulous1

[mrfabulous1]

Hi alden Hope you are also having a wonderful day!

@dalden

I am not sure how to tell which image this error is pointing towards. Is there a way to identify the issue image, or to fix all potential trouble images?

I am fairly new to both python and ML.

python PIL - check if image is transparent

Python Convert Image Background to Transparent: A Step Guide - Python Pillow...

These links should be able to help you also jimmiemunyi may be tell us how he located the the rgba files.

Cheers mrfabulous1

[jimmiemunyi]

Hope you are having a wonderful day too @mrfabulous1
Thanks for the links. I will look into them.

One last question. I did these lessons some months back and didn’t have this error. I got images and trained them without any conversions.
Why are we getting them now? Did the library change or something?

[jimmiemunyi]

@mrfabulous1

also jimmiemunyi may be tell us how he located the the rgba files.

Unfortunately I have a slower implementation where I convert all the images, without checking which ones needed fixing.
Here is my not optimized solution:

if not path.exists():
path.mkdir()
for o in bear_types:
    dest = (path/o)
    dest.mkdir(exist_ok=True)
    results = search_images_ddg_corrected(f'{o} bear')
    download_images(dest, urls=results)

    # remove any png images to avoid some errors
    # [os.remove(path/o/file) for file in os.listdir(path/o) if file.endswith('.png')]

    # convert all images to RGBA
    for image in os.listdir(path/o):
      im = Image.open(image)
      im.convert("RGBA").save(f"{image}2.png")

Replace this code (or something along those lines) in the loop of downloading the bear images

[mgloria (gloria macia)]

Maybe you fancy now trying to deploy it. I just managed to do it after overcoming a few beginner mistakes. This could be useful to you and others!

[mrfabulous1]

Hi jimmiemunyi hope your having a beautiful day!

Love your solution, any solution that moves one forward is a good one!

Cheers mrfabulous1

[mrfabulous1]

Hi immiemunyi hope your having a marvelous day!

@jimmiemunyi

One last question. I did these lessons some months back and didn’t have this error. I got images and trained them without any conversions.
Why are we getting them now? Did the library change or something?

The greatest challenge for me in this whole ML, AI, Computing domain, as a whole is that everything is changing and quickly!

• Libraries change
• Versions change
• Parts of all libraries are deprecated
• The OS changes
• Websites change
• Fastai changes continuously! (Look on Git)
• Small changes in the libraries are made daily
• In some cases small changes don’t have a version number and are often unintended side effects of other changes.
• Images from search engines can have their formats changed
• Browsers/websites change the way/format data is downloaded (e.g. try saving a medium.com page).
• Libraries can change anywhere in your echo system. (eg. cloud notebook provider, any apps you use)

I have experienced all the above regularly

It can be difficult to keep up with every change!

In answer to your question its been my experience that every part of the of the solution changes continuously. (we tend to notice the ones that break our system more!)

Antidote (partial)

• Virtual environments
• Archive data used (at least some samples)
• Record library versions.
• Keep evidence of results
• Save date app last seen working

I’m still learning so I have to do the above, currently 50% of the time I spend is fixing something that is broken or that I don’t quite understand.

Hope this helps.
Cheers mrfabulous1

[jimmiemunyi]

Hello @mrfabulous1,

Yes, I experience the above issues too. I guess its part of the job
Thanks for your effort to help with debugging issues

[taunoerik]

I changed the code to check which ones need colour mode repairing.

if not path.exists():
path.mkdir()
for o in bear_types:
dest = (path/o)
dest.mkdir(exist_ok=True)
results = search_images_bing(key, f’{o} bear’)
download_images(dest, urls=results.attrgot(‘contentUrl’))

    # convert images to RGBA
    for image in os.listdir(path/o):
      im = Image.open(image)
      # If is png image
      if im.format is 'PNG':
        # and is not RGBA
        if im.mode is not 'RGBA':
          im.convert("RGBA").save(f"{image}2.png")

[ckehl (Chris)]

@taunoerik

    # convert images to RGBA
    for image in os.listdir(path/o):
      im = Image.open(image)
      # If is png image
      if im.format is 'PNG':
        # and is not RGBA
        if im.mode is not 'RGBA':
          im.convert("RGBA").save(f"{image}2.png")

@taunoerik Thanks for the share. That worked great for me. I actually just wanted to hide or log the UserWarning, but it wouldn’t log for me.

[jdf001]

I got error when running the code:
for idx,cat in cleaner.change(): shutil.move(str(cleaner.fns[idx]), path/cat)
May I know what does that mean?

---

Error Traceback (most recent call last)
in ()
----> 1 for idx,cat in cleaner.change(): shutil.move(str(cleaner.fns[idx]), path/cat)

/usr/lib/python3.7/shutil.py in move(src, dst, copy_function)
562 real_dst = os.path.join(dst, _basename(src))
563 if os.path.exists(real_dst):
→ 564 raise Error(“Destination path ‘%s’ already exists” % real_dst)
565 try:
566 os.rename(src, real_dst)

Error: Destination path ‘bears/grizzly/Image_13.jpg’ already exists

[ckehl (Chris)]

Thanks, where in the code did you put this? I’ll try adding it to the beginning where I create the folders.

[patimejia]

quick & easy fix:

To fix the Palette image or Alpha Channel Error, I added the import before running the learner:

from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True 

then run:

• learn = vision_learner(dls, resnet18, metrics=error_rate)
• learn.fine_tune(4)

What are the potential drawbacks of using ImageFile.LOAD_TRUNCATED_IMAGES = True to handle truncated images in PIL library?

Thanks!