Is there a generic Heatmap/Grad-cam API?

Is there an API in the fastai library to generate heatmaps for a single/series of test images? I searched in the code and documentation but only find the heatmap implemented in Interpretation class function plot_top_losses.

I can probably modify the grad-cam code/projects shared on forum to meet my needs or do it from scratch as shown in lesson 6 but wanted to check if a lazy/easy way existed!

Thanks for helping!

2 Likes

Yes there is. I refactored the code a while back so that you can access GradCAM as a method of the Interpretation class:

You can pass in the dataset type and the index of the image you want.

3 Likes

This works on Learner/Interp class, so does that means I need to have the test images passed into the learner via a databunch and then run this per image passing in its test.dl index?

I was looking to invoke a function on a single image directly, similar to learn.predict. But I can try this for now. Thanks!

1 Like

I will try to implement something like this soon. But if you really want this kind of feature, you can easily monkey-patch it to do it.

1 Like

Would also be interested in having a function to get a heatmap for a single image.

Is there any follow up on this? I’m curious if after doing a learn.predict(img) following the Inference Learning docs, can we apply a heat map over the test image to visualize which pixels contributed most to the prediction?

1 Like

@rdass Yes in fact I did refactor the original GradCAM code a while back and added it to the library.

See the library code here.

You can pass in the index of the dataset that you want to view the GradCAM image for.

2 Likes

@ilovescience thanks for reply. I must be doing something stupid or I don’t fully understand how to apply the code you linked to…

Here’s my situation:

  1. Test image (no label) that I want to classify based on a fine-tuned model trained on a similar dataset as the test image.
    test_img = test_img.resize(torch.Size([test_img.shape[0],224, 224]))
    test_img.data.shape
    torch.Size([3, 224, 224])

  2. Load an inference learner
    learned_model = load_learner(path, ‘export_models/new_model_weights.pkl’)

  3. Predict classification (CORRECT!)
    learned_model.predict(test_img)
    (Category class 3,
    tensor(3),
    tensor([0.0157, 0.0804, 0.0055, 0.8985]))

  4. To see the corresponding heat map:
    interp = ClassificationInterpretation.GradCAM(learned_model, test_img)


AttributeError Traceback (most recent call last)
in
----> 1 interp = ClassificationInterpretation.GradCAM(learn_se_rn50, test[0])

/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/fastai/vision/learner.py in _cl_int_gradcam(self, idx, ds_type, heatmap_thresh, image)
137
138 def _cl_int_gradcam(self, idx, ds_type:DatasetType=DatasetType.Valid, heatmap_thresh:int=16, image:bool=True):
–> 139 m = self.learn.model.eval()
140 im,cl = self.learn.data.dl(ds_type).dataset[idx]
141 cl = int(cl)

AttributeError: ‘Learner’ object has no attribute ‘learn’

Hello sorry for the late response.

Two things:

  1. Create an instance of ClassificationInterpretation first then call GradCAM:
interp = ClassificationInterpretation.from_learner(learned_model)
gradcam_image = interp.GradCAM(0,ds_type=DatasetType.Test)
  1. Actually the index of the dataset needs to be provided along with the dataset type as shown above. So the best way to do this is to add the image to the test dataset then get that index and pass it to GradCAM. I realize now this is clearly not the most efficient way of doing this, and if the correct functionality is not yet implemented in fastai2 I will add it soon…

Thanks for getting back to me, @ilovescience. I think I’m close to the solution, but still getting errors…

# Test image directory
test

ImageList (2 items)
Image (3, 299, 299),Image (3, 299, 299)

# Inference model
learned_model1 = load_learner(path, 'export_models/model1.pkl', test=test)

# get predictions
preds,y = learned_model1.get_preds(ds_type=DatasetType.Test)
preds

tensor([[7.0309e-01, 3.0540e-02, 2.6613e-01, 2.3129e-04],
[5.3039e-02, 5.6551e-01, 3.5939e-03, 3.7785e-01]])

# Load interpretation class - no errors (success, I think?)
interp = ClassificationInterpretation.from_learner(learned_model1, ds_type=DatasetType.Test, tta=True)

# Plot confusion matrix - works!
interp.plot_confusion_matrix()

# Plot gradcam image (as per your suggestion) - error :( 
gradcam_image = interp.GradCAM(0,ds_type=DatasetType.Test)

TypeError Traceback (most recent call last)
in
----> 1 gradcam_image = interp.GradCAM(0, ds_type=DatasetType.Test, tta=True)

TypeError: _cl_int_gradcam() got an unexpected keyword argument ‘ds_type’

# Thought maybe the plot_top_loss() might work?
interp.plot_top_losses(2, heatmap=True)

--------------------------------------------------------------------------- TypeError Traceback (most recent call last) in ----> 1 interp.plot_top_losses(2, figsize=(15,11), heatmap=True) /exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/fastai/vision/learner.py in _cl_int_plot_top_losses(self, k, largest, figsize, heatmap, heatmap_thresh, return_fig) 174 for i,idx in enumerate(tl_idx): 175 im,cl = self.data.dl(self.ds_type).dataset[idx] --> 176 cl = int(cl) 177 im.show(ax=axes.flat[i], title= 178 f’{classes[self.pred_class[idx]]}/{classes[cl]} / {self.losses[idx]:.2f} / {self.preds[idx][cl]:.2f}’) TypeError: int() argument must be a string, a bytes-like object or a number, not ‘EmptyLabel’