Hey, sorry for the late reply. If you are using the latest version of fastai lib, the ‘probs’ attribute from ClassificationInterpretation is replaced with ‘preds’. Changing interp.probs[…] to interp.preds[…] should make the code work again (line 20). Let me know since I can’t test it right now.
1 Like
Hi @quan.tran,
I got the heat maps after doing the changes you suggested. It worked as expected. Thanks for the reply.
1 Like
Thanks for the reply, Quan. I’ll have a go at it.
1 Like
coming late to the game , but just wanted to tell that this is an awesome work ! thanks a lot for sharing!
1 Like
Thanks for the kind words!
Hi @quan.tran, I’m curious if we can implement the GradCam function using a load_learner(). I’m trying to show using a single test image, which pixels fired via Inference Learning. Below is an overall approach that I followed using your very helpful notebook. I’m getting a NameError despite being able to load both ‘gc’ and ‘gradcam’ libraries with no problems:
import fastai.version
print(fastai.__version__)
print(torch.__version__)
1.0.57
1.1.0
import gc
from gradcam import *
img = path/'test_img.jpg'
learn = load_learner(path, 'weight_file.pkl')
type(learn)
type(img)
fastai.basic_train.Learner
fastai.vision.image.Image
gcam = GradCam.from_one_img(learn, img)
NameError: name 'GradCam' is not defined
Any advice would be most appreciated!
Hey @rdass, I just rerun the code on my end and it seems like everything still runs fine (I am using the latest fastai 1.0.60 with torch 1.3.1) though I need to downgrade pillow to 6.1 to make from fastai.vision import * work. Are you running the code in a notebook or as a python file? If it’s the latter can you check if the gradcam.py is in the same directory?
Hi @quan.tran, sorry for the late reply. I’m running the code in a notebook. The only difference from your notebook example is that in cell 5 of the notebook, I have:
test_img = test_img.resize(torch.Size([test_img.shape[0],224, 224]))
test_img.data.shape
torch.Size([3, 224, 224])
data = None
gc.collect()
np.random.seed(42)
data = ImageDataBunch.from_folder(img_path, train=".", valid_pct=0.2,
ds_tfms=get_transforms(), size=224, num_workers=4).normalize(imagenet_stats)
learn = cnn_learner(data)
learn.load(path/'export_models/saved_model.pkl')
Strangely, I’m still getting the same error, despite successfully importing the relevant libraries. Can you please describe the structure of your train and valid datasets? Do they each contain subfolders used for labeling? Thanks!
if it’s still the ‘Name Error’ then I think it’s more of a importing problem than the dataset structure as mine are really similar to the dog breed dataset. Can you just copy everything in the gradcam.py into 1 cell of your notebook and run it again?
Hi @quan.tran, thank you for your patience. I was able to run the entire gradcam.py code in a single notebook cell without any errors. However, when testing the GradCam code for a single image, I get the following error with traceback:
type(learn)
type(test_img)
fastai.basic_train.Learner
fastai.vision.image.Image
%%time
gcam = GradCam.from_one_img(learn, test_img)
gcam.plot()
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<timed exec> in <module>
<ipython-input-49-8f4fc57aeb3a> in from_one_img(cls, learn, x_img, label1, label2)
50
51 label1_idx = learn.data.classes.index(label1)
---> 52 hmap1,xb_grad1 = get_grad_heatmap(learn,xb,label1_idx,size=xb_img.shape[-1])
53 prob1 = probs[label1_idx]
54
<ipython-input-49-8f4fc57aeb3a> in get_grad_heatmap(learn, xb, y, size)
151 m = learn.model.eval();
152 target_layer = m[0][-1][-1] # last layer of group 0
--> 153 hook_a,hook_g = hooked_backward(m,xb,target_layer,y)
154
155 target_act= hook_a.stored[0].cpu().numpy()
<ipython-input-49-8f4fc57aeb3a> in hooked_backward(m, xb, target_layer, clas)
113 with hook_output(target_layer) as hook_a: #hook at last layer of group 0's output (after bn, size 512x7x7 if resnet34)
114 with hook_output(target_layer, grad=True) as hook_g: # gradient w.r.t to the target_layer
--> 115 preds = m(xb)
116 preds[0,int(clas)].backward() # same as onehot backprop
117 return hook_a,hook_g
/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491 result = self._slow_forward(*input, **kwargs)
492 else:
--> 493 result = self.forward(*input, **kwargs)
494 for hook in self._forward_hooks.values():
495 hook_result = hook(self, input, result)
/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
90 def forward(self, input):
91 for module in self._modules.values():
---> 92 input = module(input)
93 return input
94
/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491 result = self._slow_forward(*input, **kwargs)
492 else:
--> 493 result = self.forward(*input, **kwargs)
494 for hook in self._forward_hooks.values():
495 hook_result = hook(self, input, result)
/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
90 def forward(self, input):
91 for module in self._modules.values():
---> 92 input = module(input)
93 return input
94
/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491 result = self._slow_forward(*input, **kwargs)
492 else:
--> 493 result = self.forward(*input, **kwargs)
494 for hook in self._forward_hooks.values():
495 hook_result = hook(self, input, result)
/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/conv.py in forward(self, input)
336 _pair(0), self.dilation, self.groups)
337 return F.conv2d(input, self.weight, self.bias, self.stride,
--> 338 self.padding, self.dilation, self.groups)
339
340
RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 0 does not equal 1 (while checking arguments for cudnn_convolution)@quan.tran this is a fantastic bit of work, thank you so much for sharing! Following your code, I was successfully able to get this working for my Resnet34 experiment. However, it seems that VGG_19 architecture isn’t supported by your code at this time. I’m a complete n00b at deep learning stuff, but I’m sure it has something to do with the hooks, as it’s a different architecture. I’ll have to do some research myself and see if I can get it working, as I don’t quite totally understand hooks yet. If I figure out a patch, I’ll post it here. Anyway, I’ll share the traceback below:
TypeError Traceback (most recent call last)
<ipython-input-14-f3d3840dee59> in <module>()
1 img = open_image('/content/drive/My Drive/Research/images/myimage.jpg');
----> 2 gcam = GradCam.from_one_img(learn,img)
3 gcam.plot()
1 frames
/content/animation-classification/gradcam.py in from_one_img(cls, learn, x_img, label1, label2)
50
51 label1_idx = learn.data.classes.index(label1)
---> 52 hmap1,xb_grad1 = get_grad_heatmap(learn,xb,label1_idx,size=xb_img.shape[-1])
53 prob1 = probs[label1_idx]
54
/content/animation-classification/gradcam.py in get_grad_heatmap(learn, xb, y, size)
150 xb = xb.cuda()
151 m = learn.model.eval();
--> 152 target_layer = m[0][-1][-1] # last layer of group 0
153 hook_a,hook_g = hooked_backward(m,xb,target_layer,y)
154
TypeError: 'AdaptiveAvgPool2d' object does not support indexing
If anyone has found a way around this issue, I’d super appreciate any feedback!
Hey @djpecot, sorry for the delay. My code at the moment is kinda dependent on Resnet structure, and vgg19 structure is vastly different but all you need to provide is a target layer so that the gradient can be calculated with respect to this layer. For vgg19 you can choose this layer in learn.model[0] aka the vgg body, e.g. learn.model[0][-1] is the last layer of the body (AdaptiveAvgPool2d) or learn.model[0][0][52] is the layer right before the AdaptiveAvgPool2d. In fact you have 53 layers to do some experiments on 
To fix the error above you need to go into gradcam.py and manually edit the line at 152 by putting in the layer of your choice, e.g. target_layer = m[0][-1] or target_layer = m[0][0][52]. It works on my end so I bet it works on yours too.
Got it working! looks like the Densenet structure is indexed differently then Resnet. changing line 152 to target_layer = m[0][-1] nailed it. Thanks for your help! After reading the GradCAM paper, I see what you mean about having all the layers to experiment with.
1 Like
Is GRADCAM useful tool, in context to medical imaging?
To rephrase my question - often in medical imaging- a very small fraction of image is responsible for the findings, but GRADCAM or its later versions show a big red patch, including much larger areas.
This makes me wonder, if GRADCAM can help us pin point small image features that are crucial to class labelling, especillay in medical imaging.
It really depends on how the model behaves. The way deep learning model determines parts of the image that is responsible for the findings might be different from human’s who have domain knowledge. If we carefully pick our dataset and try to avoid data leak (so that the model cannot use anything other information rather than the crucial region you are talking about) then I think the model should only use that region and GRADCAM should be able to pin point it.
hey, @quan.tran can we use your code for the object detection library of fastai, if so can you please share the code for retinanet gradcam implementation?
Hi,I am a newbie and need help in implementation of your library. Since the link is not working-https://quantran.xyz/blog/building-an-image-classification-model-from-a-to-z/ ,i have a problem in passing the ‘image_idx’ while calling GradCam . Kindly provide a clear example to use it.
Regards
how to access the last layer for effnet b5 model to apply gradcam on it.from fastai.callbacks.hooks import *
hook into forward pass
def hooked_backward(m, oneBatch, cat):
# we hook into the convolutional part = m[0] of the model
# Access the second-to-last layer
layer_to_hook = m[0]._conv_head
#last_layer = m[0]._blocks[-1]
with hook_output( layer_to_hook) as hook_a:
with hook_output( layer_to_hook , grad=True) as hook_g:
preds = m(oneBatch)
preds[0,int(cat)].backward()
return hook_a,hook_g