Is there a generic Heatmap/Grad-cam API?

nbharatula · October 25, 2019, 7:38pm

Is there an API in the fastai library to generate heatmaps for a single/series of test images? I searched in the code and documentation but only find the heatmap implemented in Interpretation class function plot_top_losses.

I can probably modify the grad-cam code/projects shared on forum to meet my needs or do it from scratch as shown in lesson 6 but wanted to check if a lazy/easy way existed!

Thanks for helping!

ilovescience · October 25, 2019, 8:19pm

Yes there is. I refactored the code a while back so that you can access GradCAM as a method of the Interpretation class:

github.com

fastai/fastai/blob/c5b2faca1710c66a2bccb73becf60b14ea535b39/fastai/vision/learner.py#L138-L161


def _cl_int_gradcam(self, idx, ds_type:DatasetType=DatasetType.Valid, heatmap_thresh:int=16, image:bool=True):
    m = self.learn.model.eval()
    im,cl = self.learn.data.dl(ds_type).dataset[idx]
    cl = int(cl)
    xb,_ = self.data.one_item(im, detach=False, denorm=False) #put into a minibatch of batch size = 1
    with hook_output(m[0]) as hook_a: 
        with hook_output(m[0], grad=True) as hook_g:
            preds = m(xb)
            preds[0,int(cl)].backward() 
    acts  = hook_a.stored[0].cpu() #activation maps
    if (acts.shape[-1]*acts.shape[-2]) >= heatmap_thresh:
        grad = hook_g.stored[0][0].cpu()
        grad_chan = grad.mean(1).mean(1)
        mult = F.relu(((acts*grad_chan[...,None,None])).sum(0))
        if image:
            xb_im = Image(xb[0])
            _,ax = plt.subplots()
            sz = list(xb_im.shape[-2:])
            xb_im.show(ax,title=f"pred. class: {self.pred_class[idx]}, actual class: {self.learn.data.classes[cl]}")
            ax.imshow(mult, alpha=0.4, extent=(0,*sz[::-1],0),

This file has been truncated. show original

You can pass in the dataset type and the index of the image you want.

nbharatula · October 25, 2019, 8:27pm

This works on Learner/Interp class, so does that means I need to have the test images passed into the learner via a databunch and then run this per image passing in its test.dl index?

I was looking to invoke a function on a single image directly, similar to learn.predict. But I can try this for now. Thanks!

ilovescience · October 26, 2019, 1:47am

I will try to implement something like this soon. But if you really want this kind of feature, you can easily monkey-patch it to do it.

Seesam · November 17, 2019, 8:06pm

Would also be interested in having a function to get a heatmap for a single image.

rdass · February 14, 2020, 1:15am

Is there any follow up on this? I’m curious if after doing a learn.predict(img) following the Inference Learning docs, can we apply a heat map over the test image to visualize which pixels contributed most to the prediction?

ilovescience · February 14, 2020, 2:05am

@rdass Yes in fact I did refactor the original GradCAM code a while back and added it to the library.

See the library code here.

You can pass in the index of the dataset that you want to view the GradCAM image for.

rdass · February 14, 2020, 8:59pm

@ilovescience thanks for reply. I must be doing something stupid or I don’t fully understand how to apply the code you linked to…

Here’s my situation:

Test image (no label) that I want to classify based on a fine-tuned model trained on a similar dataset as the test image.
test_img = test_img.resize(torch.Size([test_img.shape[0],224, 224]))
test_img.data.shape
torch.Size([3, 224, 224])
Load an inference learner
learned_model = load_learner(path, ‘export_models/new_model_weights.pkl’)
Predict classification (CORRECT!)
learned_model.predict(test_img)
(Category class 3,
tensor(3),
tensor([0.0157, 0.0804, 0.0055, 0.8985]))
To see the corresponding heat map:
interp = ClassificationInterpretation.GradCAM(learned_model, test_img)

AttributeError Traceback (most recent call last)
in
----> 1 interp = ClassificationInterpretation.GradCAM(learn_se_rn50, test[0])

/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/fastai/vision/learner.py in _cl_int_gradcam(self, idx, ds_type, heatmap_thresh, image)
137
138 def _cl_int_gradcam(self, idx, ds_type:DatasetType=DatasetType.Valid, heatmap_thresh:int=16, image:bool=True):
–> 139 m = self.learn.model.eval()
140 im,cl = self.learn.data.dl(ds_type).dataset[idx]
141 cl = int(cl)

AttributeError: ‘Learner’ object has no attribute ‘learn’

ilovescience · February 18, 2020, 2:29am

Hello sorry for the late response.

Two things:

Create an instance of ClassificationInterpretation first then call GradCAM:

interp = ClassificationInterpretation.from_learner(learned_model)
gradcam_image = interp.GradCAM(0,ds_type=DatasetType.Test)

Actually the index of the dataset needs to be provided along with the dataset type as shown above. So the best way to do this is to add the image to the test dataset then get that index and pass it to GradCAM. I realize now this is clearly not the most efficient way of doing this, and if the correct functionality is not yet implemented in fastai2 I will add it soon…

rdass · February 18, 2020, 2:45am

Thanks for getting back to me, @ilovescience. I think I’m close to the solution, but still getting errors…

# Test image directory
test

ImageList (2 items)
Image (3, 299, 299),Image (3, 299, 299)

# Inference model
learned_model1 = load_learner(path, 'export_models/model1.pkl', test=test)

# get predictions
preds,y = learned_model1.get_preds(ds_type=DatasetType.Test)
preds

tensor([[7.0309e-01, 3.0540e-02, 2.6613e-01, 2.3129e-04],
[5.3039e-02, 5.6551e-01, 3.5939e-03, 3.7785e-01]])

# Load interpretation class - no errors (success, I think?)
interp = ClassificationInterpretation.from_learner(learned_model1, ds_type=DatasetType.Test, tta=True)

# Plot confusion matrix - works!
interp.plot_confusion_matrix()

# Plot gradcam image (as per your suggestion) - error :( 
gradcam_image = interp.GradCAM(0,ds_type=DatasetType.Test)

TypeError Traceback (most recent call last)
in
----> 1 gradcam_image = interp.GradCAM(0, ds_type=DatasetType.Test, tta=True)

TypeError: _cl_int_gradcam() got an unexpected keyword argument ‘ds_type’

# Thought maybe the plot_top_loss() might work?
interp.plot_top_losses(2, heatmap=True)

--------------------------------------------------------------------------- TypeError Traceback (most recent call last) in ----> 1 interp.plot_top_losses(2, figsize=(15,11), heatmap=True) /exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/fastai/vision/learner.py in _cl_int_plot_top_losses(self, k, largest, figsize, heatmap, heatmap_thresh, return_fig) 174 for i,idx in enumerate(tl_idx): 175 im,cl = self.data.dl(self.ds_type).dataset[idx] --> 176 cl = int(cl) 177 im.show(ax=axes.flat[i], title= 178 f’{classes[self.pred_class[idx]]}/{classes[cl]} / {self.losses[idx]:.2f} / {self.preds[idx][cl]:.2f}’) TypeError: int() argument must be a string, a bytes-like object or a number, not ‘EmptyLabel’

alejo101010 · September 24, 2020, 1:43am

Hi rdass,

Can you help me? I have the same problem

Thanks