Upload own image fail. OSError: Error handling image


#1

Hello

I downloaded some images and uploaded according to the format as given in lesson1.ipynb. The code reads and plots the images but the learning step right after the plot failed. The following error came up during the step:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/fastai/courses/dl1/fastai/dataset.py in open_image(fn)
    288             else:
--> 289                 im = cv2.imread(str(fn), flags).astype(np.float32)/255
    290             if im is None: raise OSError(f'File not recognized by opencv: {fn}')

AttributeError: 'NoneType' object has no attribute 'astype'

The above exception was the direct cause of the following exception:

OSError                                   Traceback (most recent call last)
<ipython-input-80-e6c87b20ce86> in <module>()
      1 arch=resnet34
      2 data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
----> 3 learn = ConvLearner.pretrained(arch, data, precompute=True)
      4 learn.fit(0.01, 2)

~/fastai/courses/dl1/fastai/conv_learner.py in pretrained(cls, f, data, ps, xtra_fc, xtra_cut, custom_head, precompute, pretrained, **kwargs)
    112         models = ConvnetBuilder(f, data.c, data.is_multi, data.is_reg,
    113             ps=ps, xtra_fc=xtra_fc, xtra_cut=xtra_cut, custom_head=custom_head, pretrained=pretrained)
--> 114         return cls(data, models, precompute, **kwargs)
    115 
    116     @classmethod

~/fastai/courses/dl1/fastai/conv_learner.py in __init__(self, data, models, precompute, **kwargs)
     98         if hasattr(data, 'is_multi') and not data.is_reg and self.metrics is None:
     99             self.metrics = [accuracy_thresh(0.5)] if self.data.is_multi else [accuracy]
--> 100         if precompute: self.save_fc1()
    101         self.freeze()
    102         self.precompute = precompute

~/fastai/courses/dl1/fastai/conv_learner.py in save_fc1(self)
    177         m=self.models.top_model
    178         if len(self.activations[0])!=len(self.data.trn_ds):
--> 179             predict_to_bcolz(m, self.data.fix_dl, act)
    180         if len(self.activations[1])!=len(self.data.val_ds):
    181             predict_to_bcolz(m, self.data.val_dl, val_act)

~/fastai/courses/dl1/fastai/model.py in predict_to_bcolz(m, gen, arr, workers)
     15     lock=threading.Lock()
     16     m.eval()
---> 17     for x,*_ in tqdm(gen):
     18         y = to_np(m(VV(x)).data)
     19         with lock:

~/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py in __iter__(self)
    977 """, fp_write=getattr(self.fp, 'write', sys.stderr.write))
    978 
--> 979             for obj in iterable:
    980                 yield obj
    981                 # Update and possibly print the progressbar.

~/fastai/courses/dl1/fastai/dataloader.py in __iter__(self)
     86                 # avoid py3.6 issue where queue is infinite and can result in memory exhaustion
     87                 for c in chunk_iter(iter(self.batch_sampler), self.num_workers*10):
---> 88                     for batch in e.map(self.get_batch, c):
     89                         yield get_tensor(batch, self.pin_memory, self.half)
     90 

~/anaconda3/lib/python3.6/concurrent/futures/_base.py in result_iterator()
    584                     # Careful not to keep a reference to the popped future
    585                     if timeout is None:
--> 586                         yield fs.pop().result()
    587                     else:
    588                         yield fs.pop().result(end_time - time.time())

~/anaconda3/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

~/anaconda3/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

~/anaconda3/lib/python3.6/concurrent/futures/thread.py in run(self)
     54 
     55         try:
---> 56             result = self.fn(*self.args, **self.kwargs)
     57         except BaseException as exc:
     58             self.future.set_exception(exc)

~/fastai/courses/dl1/fastai/dataloader.py in get_batch(self, indices)
     73 
     74     def get_batch(self, indices):
---> 75         res = self.np_collate([self.dataset[i] for i in indices])
     76         if self.transpose:   res[0] = res[0].T
     77         if self.transpose_y: res[1] = res[1].T

~/fastai/courses/dl1/fastai/dataloader.py in <listcomp>(.0)
     73 
     74     def get_batch(self, indices):
---> 75         res = self.np_collate([self.dataset[i] for i in indices])
     76         if self.transpose:   res[0] = res[0].T
     77         if self.transpose_y: res[1] = res[1].T

~/fastai/courses/dl1/fastai/dataset.py in __getitem__(self, idx)
    201             xs,ys = zip(*[self.get1item(i) for i in range(*idx.indices(self.n))])
    202             return np.stack(xs),ys
--> 203         return self.get1item(idx)
    204 
    205     def __len__(self): return self.n

~/fastai/courses/dl1/fastai/dataset.py in get1item(self, idx)
    194 
    195     def get1item(self, idx):
--> 196         x,y = self.get_x(idx),self.get_y(idx)
    197         return self.get(self.transform, x, y)
    198 

~/fastai/courses/dl1/fastai/dataset.py in get_x(self, i)
    298         super().__init__(transform)
    299     def get_sz(self): return self.transform.sz
--> 300     def get_x(self, i): return open_image(os.path.join(self.path, self.fnames[i]))
    301     def get_n(self): return len(self.fnames)
    302 

~/fastai/courses/dl1/fastai/dataset.py in open_image(fn)
    291             return cv2.cvtColor(im, cv2.COLOR_BGR2RGB)
    292         except Exception as e:
--> 293             raise OSError('Error handling image at: {}'.format(fn)) from e
    294 
    295 class FilesDataset(BaseDataset):

OSError: Error handling image at: ../../../data/orange-durian/train/orange/11.jpg

Has anyone had this problem?

Cheers


#2

Hi

I followed this Failing to complete training in Lesson 1 for own data

removed the problematic pictures and the code runs. However, how do I inspect the image (without seeing the image) so that I know what kind of images are allowed etc.
I wouldn’t want to have to throw away bad images everytime this comes up.How can I inspect the images?


(Jonathan Aghachi) #3

Hey Hud,

I was having the same issue using the script to download images from google.
When I reviewed the pictures, I noticed they were just bad image every time. I got tired of having to do this every time I tested on different data sets downloaded, so I created this quick “bad image deleter”… this will at least report back to you the images that weren’t able to be opened and were deleted.

WARNING: i commented out the part that deletes pictures for you. So it’ll just spit out the images in question in a list. If you want it to delete just remove the ‘#’ in front of the “os.remove” line. If you do that, don’t use this on a dataset that you didn’t back up because it’ll automatically delete the pictures

imagesL = {}
printIt = []

for foldername in data.classes:
    imagesL.update({foldername : os.listdir("data/images/" + foldername)})
for key in imagesL:
    for image in imagesL[key]:
            try:
                img = open_image("data/images/" + key + "/" + image) 
            except OSError:
                printIt.append("deleting " + "data/images/" + key + "/" + image + str(OSError))
               # os.remove("data/images/" + key + "/" + image, dir_fd=None)
print(printIt)

Let me know if you need some help!


#4

Hey thanks for this. I’ll get this in my code and update you if it works or I have problems.


(Jonathan Aghachi) #5

hud looks like i might have created a redundant function.

fastai’s “verify_images” is supposed to do this. I’m not sure at the moment how come it didn’t work for me, i’ll look into it.

doc(verify_images)

run that to learn more.

takes a delete argument to delete unopenable files.
example:

verify_images(pathI/c, delete=True, max_workers=8)

(Jonathan Aghachi) #6

ah i found it, i’m such a dummy! my folder names were not set correctly, therefore not finding any pictures to delete! so dont use my messy function, use

verify_images()

function and set “delete=True” as one of the arguments.


(James_Ying) #7

Thanks, met same problem. your answer is very helpful.