IndexError: index 701921 is out of bounds for axis 0 with size 314

I am trying to interpret top losses from a model that I successfully built and keep on getting the following error. I found a couple of links with this topic but with no real followup.

Code
interp.plot_top_losses(2, figsize=(15,11))

Error

IndexError                                Traceback (most recent call last)
<ipython-input-42-ad32031f46cf> in <module>()
----> 1 interp.plot_top_losses(2, figsize=(15,11))

/home/gbogu17/.local/lib/python3.7/site-packages/fastai/vision/learner.py in _cl_int_plot_top_losses(self, k, largest, figsize, heatmap, heatmap_thresh, return_fig)
    173     fig.suptitle('prediction/actual/loss/probability', weight='bold', size=14)
    174     for i,idx in enumerate(tl_idx):
--> 175         im,cl = self.data.dl(self.ds_type).dataset[idx]
    176         cl = int(cl)
    177         im.show(ax=axes.flat[i], title=

/home/gbogu17/.local/lib/python3.7/site-packages/fastai/data_block.py in __getitem__(self, idxs)
    646         idxs = try_int(idxs)
    647         if isinstance(idxs, Integral):
--> 648             if self.item is None: x,y = self.x[idxs],self.y[idxs]
    649             else:                 x,y = self.item   ,0
    650             if self.tfms or self.tfmargs:

/home/gbogu17/.local/lib/python3.7/site-packages/fastai/data_block.py in __getitem__(self, idxs)
    116         "returns a single item based if `idxs` is an integer or a new `ItemList` object if `idxs` is a range."
    117         idxs = try_int(idxs)
--> 118         if isinstance(idxs, Integral): return self.get(idxs)
    119         else: return self.new(self.items[idxs], inner_df=index_row(self.inner_df, idxs))
    120 

/home/gbogu17/.local/lib/python3.7/site-packages/fastai/vision/data.py in get(self, i)
    268 
    269     def get(self, i):
--> 270         fn = super().get(i)
    271         res = self.open(fn)
    272         self.sizes[i] = res.size

/home/gbogu17/.local/lib/python3.7/site-packages/fastai/data_block.py in get(self, i)
     72     def get(self, i)->Any:
     73         "Subclass if you want to customize how to create item `i` from `self.items`."
---> 74         return self.items[i]
     75     def __repr__(self)->str:
     76         items = [self[i] for i in range(min(5,len(self.items)))]

IndexError: index 701921 is out of bounds for axis 0 with size 314

I had the same error from the interp, then I realized that I am trying it for a segmentation learner, and unfortunately, plot_top_losses is actually is for classification https://github.com/fastai/fastai/blob/master/fastai/vision/learner.py#L163.

1 Like

How would you go about using something similar for getting top losses then creating a data bunch from the result for more training?

There is the top_losses method in the base class Interpretation: https://github.com/fastai/fastai/blob/284ccb3e43178c013e8bd181ebb808490bf67e0e/fastai/train.py#L163

However instead of using this method, there is also the ds argument (self.ds). I think this is the validation ds by default. So I think thresholding the self.losses to get the indexes for the self.ds might work. What do you think?

Did you get any solution to this problem. I am getting the same error. But irony is when I create a data bunch using ā€œImageList.from_df or ImageList.from_csvā€ it gives me this error but when I use ā€œImageList.from_folderā€ it works fine.

I have tried to debug a bit, the problem seems to be with the losses. while preds are returned only for the validation set. losses are returned for entire training set which is making it go out of range.

Please share if anyone figured this out. Thanks