BrokenProcessPool exception on running function verify_images

I am currently on Lesson 2 of Deep learning for coders(2020). Upon running the verify_images function, as stated in the lesson’s notebook:

I get a BrokenProcessPool exception.

This error could easily be corrected in FastAi v1, by specifying max_workers to 1. However, there is no such argument available(max_workers) in the current version of FastAi (for function verify_images) . Is there any work around for this?

2 Likes

Loop through them yourself with verify_image

Sure thing, but for applying Path.unlink , i would need to map it to an object having truth values for each image. And, as Jeremy said, the verify_images returns an L object ( midway between a python list and a numpy array). I am not sure how to make and add elements to such an object.

unlink is a pathlib function. I’m not sure what they actually added to L as I haven’t started 2020 yet but all you need to do is loop through fns and call verify_image. By the looks of the list comprehension in the call stack I’d say it will return None if the image is invalid (you can test this by renaming a text file test.jpg and call it to see what it does). Then call unlink to delete if it’s invalid.

Also if this happens again it’s far more useful to us if you cut and paste the entire call stack, don’t screen shot the top part, the useful info is normally right at (or near) the bottom of it.

Got this working for a temporary fix:

for fn in fns:
    passed = verify_image(fn)
    if not passed:
        os.unlink(fn)

although, they should bring back an argument for setting max_workers in verify_images.

Okay! Thanks for your help.

I get the following error on running unlink-

PermissionError: [WinError 5] Access is denied:

I ran the notebook as administrator but still got the error

Make sure the path you feed to unlink is that of an image and not of a folder! Here’s a snap of my notebook :


Hope this helps!

Jupyter + Windows = Broken Multithreading

Alternatively you can redefine verify_images to be single-threaded, in a cell just before execution.

def verify_images(fns):
    "Find images in `fns` that can't be opened"
    return L(fns[i] for i,o in enumerate(fns.map(verify_image)) if not o)
3 Likes