Problem with ClassificationInterpretation and ImageClassifierCleaner

Hi there !

ClassificationInterpretation and ImageClassifierCleaner make the kernel crash all the time. The problem is I am not getting any error… The kernel just crashes while the learner iterates through the files…

Any idea why that might be the case ?

edit:
note that the model fine_tunes without any problem, which I believe removes the possibility that a file might be corrupt. Also, I tried changing the batch_size to no avail. Finally, show_results() works without problem as well… very curious… maybe my hardware is screwed somehow ?

I’ve kept investigating this a little. I’ve taken 1/10th of the data, tried ClassificationInterpretation and ImageClassifierCleaner and they both worked. So I did that for the rest of the data, 1/10th at a time. Everything worked perfectly…
@sgugger sorry for the at-mentionning, but I believe it is possible that there is something that causes OOM errors with those two things (both have in common that they make predictions on the entire datasets; since the transforms happen on the GPU before prediction (I believe), maybe this is where things crash ?)

edit: also, I tried to use datablock.dataloaders() method with the ‘sample’ and ‘n’ parameters to see if I could take only a few items in the folders without having to reorganize them. Neither seemed to work. Maybe I got their intended use wrong, but in that case maybe something to do just that should exist ( i.e you have two folders with your classes, but would like at first to run a quick model on only ten% of available files for each class).

If the kernel crashes, it means Python has crashed. Is there an error message in the shell window that you launched Jupyter from?

If you have lots and lots of pictures, it’s possible you kernel crashes because it goes out of RAM. You should try applying it to a smaller dataloader.

1 Like

This definitely seems to be worse in fastai2 - I suspect due to it storing inputs now too. Altering the function not to store inputs, as I don’t need this, uses orders of magnitude less memory:

@classmethod
def from_learner(cls, learn, ds_idx=1, dl=None, act=None):
    "Construct interpretatio object from a learner"
    if dl is None: dl = learn.dls[ds_idx]
    preds, targs, decoded, losses = learn.get_preds(dl=dl, with_input=False, with_loss=True, with_decoded=True, act=None)
    return cls(dl, [], preds, targs, decoded, losses)
3 Likes

Thanks @aaerox - that worked for me.

To add a little context: I was noticing that while ClassificationInterpretation.from_learner was running, my RAM usage was slowly increasing until it maxed out and the kernel died.

1 Like

While using classificationInterpretation my notebook crashed due to high ram consumption. How to solve this issue.

1 Like

I’m confused. If this doesn’t happen during training why would it happen when doing an interpretation on what is probably only 20% of the training data? Does a bug report need to be submitted or is this in a queue already? The reason I ask is I am using the latest version and it appears to have an issue with RAM usage.