GradCam and Guided Backprop intergration in Fastai library

@quan.tran this is a fantastic bit of work, thank you so much for sharing! Following your code, I was successfully able to get this working for my Resnet34 experiment. However, it seems that VGG_19 architecture isn’t supported by your code at this time. I’m a complete n00b at deep learning stuff, but I’m sure it has something to do with the hooks, as it’s a different architecture. I’ll have to do some research myself and see if I can get it working, as I don’t quite totally understand hooks yet. If I figure out a patch, I’ll post it here. Anyway, I’ll share the traceback below:

TypeError                                 Traceback (most recent call last)
<ipython-input-14-f3d3840dee59> in <module>()
      1 img = open_image('/content/drive/My Drive/Research/images/myimage.jpg');
----> 2 gcam = GradCam.from_one_img(learn,img)
      3 gcam.plot()

1 frames
/content/animation-classification/gradcam.py in from_one_img(cls, learn, x_img, label1, label2)
     50 
     51         label1_idx = learn.data.classes.index(label1)
---> 52         hmap1,xb_grad1 = get_grad_heatmap(learn,xb,label1_idx,size=xb_img.shape[-1])
     53         prob1 = probs[label1_idx]
     54 

/content/animation-classification/gradcam.py in get_grad_heatmap(learn, xb, y, size)
    150     xb = xb.cuda()
    151     m = learn.model.eval();
--> 152     target_layer = m[0][-1][-1] # last layer of group 0
    153     hook_a,hook_g = hooked_backward(m,xb,target_layer,y)
    154 

TypeError: 'AdaptiveAvgPool2d' object does not support indexing

If anyone has found a way around this issue, I’d super appreciate any feedback!

Hey @djpecot, sorry for the delay. My code at the moment is kinda dependent on Resnet structure, and vgg19 structure is vastly different but all you need to provide is a target layer so that the gradient can be calculated with respect to this layer. For vgg19 you can choose this layer in learn.model[0] aka the vgg body, e.g. learn.model[0][-1] is the last layer of the body (AdaptiveAvgPool2d) or learn.model[0][0][52] is the layer right before the AdaptiveAvgPool2d. In fact you have 53 layers to do some experiments on :slight_smile:

To fix the error above you need to go into gradcam.py and manually edit the line at 152 by putting in the layer of your choice, e.g. target_layer = m[0][-1] or target_layer = m[0][0][52]. It works on my end so I bet it works on yours too.

@quan.tran Thanks for the response! I’ll tinker with it and see if I can get it working :thinking:

1 Like

Got it working! looks like the Densenet structure is indexed differently then Resnet. changing line 152 to target_layer = m[0][-1] nailed it. Thanks for your help! After reading the GradCAM paper, I see what you mean about having all the layers to experiment with.

1 Like

Is GRADCAM useful tool, in context to medical imaging?

To rephrase my question - often in medical imaging- a very small fraction of image is responsible for the findings, but GRADCAM or its later versions show a big red patch, including much larger areas.

This makes me wonder, if GRADCAM can help us pin point small image features that are crucial to class labelling, especillay in medical imaging.

It really depends on how the model behaves. The way deep learning model determines parts of the image that is responsible for the findings might be different from human’s who have domain knowledge. If we carefully pick our dataset and try to avoid data leak (so that the model cannot use anything other information rather than the crucial region you are talking about) then I think the model should only use that region and GRADCAM should be able to pin point it.

hey, @quan.tran can we use your code for the object detection library of fastai, if so can you please share the code for retinanet gradcam implementation?

Hi,I am a newbie and need help in implementation of your library. Since the link is not working-https://quantran.xyz/blog/building-an-image-classification-model-from-a-to-z/ ,i have a problem in passing the ‘image_idx’ while calling GradCam . Kindly provide a clear example to use it.
Regards

Gradcam - Chapter 18. GitHub - fastai/fastbook2e: The fastai book, 2nd edition (in progress)

1 Like

how to access the last layer for effnet b5 model to apply gradcam on it.from fastai.callbacks.hooks import *

hook into forward pass

def hooked_backward(m, oneBatch, cat):
# we hook into the convolutional part = m[0] of the model
# Access the second-to-last layer
layer_to_hook = m[0]._conv_head
#last_layer = m[0]._blocks[-1]
with hook_output( layer_to_hook) as hook_a:
with hook_output( layer_to_hook , grad=True) as hook_g:
preds = m(oneBatch)
preds[0,int(cat)].backward()
return hook_a,hook_g