GradCam and Guided Backprop intergration in Fastai library

quan.tran · December 26, 2018, 2:33am

Hi all,
I am working on an image classification (classify scenes/characters from various animated movies) for fun during the break and realize how important it is to understand and interpret what your model learns so you can make improvement on the model or on the dataset you have (so far it helps me build a solid validation set + fix a data collection error which could lead to data leakage). Based on Jeremy’s lesson 6 pet nb, henripal’s notebook and GradCam paper, I write some code to quickly generate gradcam and guided backprop, and would love to contribute it to fastai library.

Here are few examples
(Notation: Gradcam -> GC, Guided Backprop -> GBP
Images from left to right: original image / GC w.r.t Predicted label(with probability) / GBP w.r.t Predicted label / GC w.r.t Actual label (with probability) / GBP w.r.t Actual label)

Another example with only the gradcam w.r.t predicted label (programmable)

Here is the code / example and in short, you only need these lines of code.

# from ClassificationIntepretation object. 
interp = ClassificationInterpretation.from_learner(learn,ds_type = DatasetType.Valid)

gcam = GradCam.from_interp(learner,interp,image_idx) #image_idx from ds.valid_ds or ds.test_ds
gcam.plot() #plot both GradCam and GuidedBackprop. 
# You can also choose either one by passing parameters into plot function

This can be an addition to ClassificationInterpretation, e.g after interp.most_confused() and want to find out more about those most confused classes (example in the gist). You can also do gradcam + guided bp on test set by changing ds_type to DatasetType.Test (could be good for Kaggle competition)

You can also plot heatmap + gbp for 1 single Image object (see more in the notebook)

# from a single Image object. 
img = open_image(path);
gcam = GradCam.from_one_img(learn,img)
gcam.plot()

Here is the project that I built where I use GradCam to visualize the model and troubleshoot: https://quantran.xyz/blog/building-an-image-classification-model-from-a-to-z/

Let me know if this looks good and I will refactor it a bit and do a PR

mgc87 · April 30, 2019, 10:54pm

Quan-
This looks great. Would you be able to lay out how one imports your gradcam.py script, then deploy it? I think it is a bit misleading to say “you only need these lines of code” when it relies upon some dependencies. Just trying to make this more user friendly so us newbies can quickly experiment with your fine contribution!

quan.tran · May 1, 2019, 2:47pm

Hi Matt,

For local experiment, you can download the gradcam.py file to your directory and import it using from gradcam import * and then you can follow my instruction in this post for plotting. If you want to deploy it to some server, you can take a look at my deployment repo where I deploy the app to Render using Starlette.io framework: https://github.com/anhquan0412/animation-classification-deployment/tree/master/app (following this tutorial: https://towardsdatascience.com/building-web-app-for-computer-vision-model-deploying-to-production-in-10-minutes-a-detailed-ec6ac52ec7e4)

I used a simplified version of gradcam.py for deployment. This file and fastai library are the 2 deep learning library dependencies for plotting the heatmap (in server.py file)

dantheman6100 · June 17, 2019, 3:09am

Hey @quan.tran,

The code you have is awesome. It’s really helping me out. Thanks!

quan.tran · June 17, 2019, 7:43pm

No problem. Glad to help!

bjinwoopark · June 28, 2019, 6:32am

When I try and execute from gradcam import * from gradcam-usecase.ipynb, I receive an error NameError: name 'DatasetType' is not defined from lines 5 and 7 of gradcam.py. Any thoughts?

bjinwoopark · June 28, 2019, 3:21pm

Which fastai version are you using? Mine might be outdated.

bjinwoopark · June 28, 2019, 7:46pm

Sorry, solved! It was my outdated library.

quan.tran · June 28, 2019, 9:02pm

I was about to comment on the fastai version, but look like you already figured that out

rijul · July 4, 2019, 8:29am

Hi, I am trying to implement the GradCam function described here using fastai, but this code throws an error.
Its says, “ClassificationInterpretation object has no attribute probs”. Can some help me here. In the “gradcam.py” it seems its used as interp.probs[]. Thanks!

quan.tran · July 11, 2019, 9:15pm

Hey, sorry for the late reply. If you are using the latest version of fastai lib, the ‘probs’ attribute from ClassificationInterpretation is replaced with ‘preds’. Changing interp.probs[…] to interp.preds[…] should make the code work again (line 20). Let me know since I can’t test it right now.

rijul · July 15, 2019, 6:46am

Hi @quan.tran,

I got the heat maps after doing the changes you suggested. It worked as expected. Thanks for the reply.

mgc87 · July 24, 2019, 2:50am

Thanks for the reply, Quan. I’ll have a go at it.

Doursand · January 31, 2020, 9:53am

coming late to the game , but just wanted to tell that this is an awesome work ! thanks a lot for sharing!

quan.tran · January 31, 2020, 11:13am

Thanks for the kind words!

rdass · February 6, 2020, 10:35pm

Hi @quan.tran, I’m curious if we can implement the GradCam function using a load_learner(). I’m trying to show using a single test image, which pixels fired via Inference Learning. Below is an overall approach that I followed using your very helpful notebook. I’m getting a NameError despite being able to load both ‘gc’ and ‘gradcam’ libraries with no problems:

import fastai.version
print(fastai.__version__)
print(torch.__version__)
1.0.57
1.1.0

import gc
from gradcam import *

img = path/'test_img.jpg'
learn = load_learner(path, 'weight_file.pkl')
type(learn)
type(img)
fastai.basic_train.Learner
fastai.vision.image.Image

gcam = GradCam.from_one_img(learn, img)
NameError: name 'GradCam' is not defined

Any advice would be most appreciated!

quan.tran · February 7, 2020, 2:12pm

Hey @rdass, I just rerun the code on my end and it seems like everything still runs fine (I am using the latest fastai 1.0.60 with torch 1.3.1) though I need to downgrade pillow to 6.1 to make from fastai.vision import * work. Are you running the code in a notebook or as a python file? If it’s the latter can you check if the gradcam.py is in the same directory?

rdass · February 14, 2020, 1:18am

Hi @quan.tran, sorry for the late reply. I’m running the code in a notebook. The only difference from your notebook example is that in cell 5 of the notebook, I have:

test_img = test_img.resize(torch.Size([test_img.shape[0],224, 224]))
test_img.data.shape
torch.Size([3, 224, 224])

data = None
gc.collect()

np.random.seed(42)
data = ImageDataBunch.from_folder(img_path, train=".", valid_pct=0.2,
        ds_tfms=get_transforms(), size=224, num_workers=4).normalize(imagenet_stats)

learn = cnn_learner(data)
learn.load(path/'export_models/saved_model.pkl')

Strangely, I’m still getting the same error, despite successfully importing the relevant libraries. Can you please describe the structure of your train and valid datasets? Do they each contain subfolders used for labeling? Thanks!

quan.tran · February 17, 2020, 10:31am

if it’s still the ‘Name Error’ then I think it’s more of a importing problem than the dataset structure as mine are really similar to the dog breed dataset. Can you just copy everything in the gradcam.py into 1 cell of your notebook and run it again?

rdass · February 18, 2020, 12:48am

Hi @quan.tran, thank you for your patience. I was able to run the entire gradcam.py code in a single notebook cell without any errors. However, when testing the GradCam code for a single image, I get the following error with traceback:

type(learn)
type(test_img)
fastai.basic_train.Learner
fastai.vision.image.Image

 %%time
gcam = GradCam.from_one_img(learn, test_img)
gcam.plot()

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<timed exec> in <module>

<ipython-input-49-8f4fc57aeb3a> in from_one_img(cls, learn, x_img, label1, label2)
     50 
     51         label1_idx = learn.data.classes.index(label1)
---> 52         hmap1,xb_grad1 = get_grad_heatmap(learn,xb,label1_idx,size=xb_img.shape[-1])
     53         prob1 = probs[label1_idx]
     54 

<ipython-input-49-8f4fc57aeb3a> in get_grad_heatmap(learn, xb, y, size)
    151     m = learn.model.eval();
    152     target_layer = m[0][-1][-1] # last layer of group 0
--> 153     hook_a,hook_g = hooked_backward(m,xb,target_layer,y)
    154 
    155     target_act= hook_a.stored[0].cpu().numpy()

<ipython-input-49-8f4fc57aeb3a> in hooked_backward(m, xb, target_layer, clas)
    113     with hook_output(target_layer) as hook_a: #hook at last layer of group 0's output (after bn, size 512x7x7 if resnet34)
    114         with hook_output(target_layer, grad=True) as hook_g: # gradient w.r.t to the target_layer
--> 115             preds = m(xb)
    116             preds[0,int(clas)].backward() # same as onehot backprop
    117     return hook_a,hook_g

/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
     90     def forward(self, input):
     91         for module in self._modules.values():
---> 92             input = module(input)
     93         return input
     94 

/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
     90     def forward(self, input):
     91         for module in self._modules.values():
---> 92             input = module(input)
     93         return input
     94 

/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/torch/nn/modules/conv.py in forward(self, input)
    336                             _pair(0), self.dilation, self.groups)
    337         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 338                         self.padding, self.dilation, self.groups)
    339 
    340 

RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 0 does not equal 1 (while checking arguments for cudnn_convolution)