GradCam and Guided Backprop intergration in Fastai library

(Quan Tran) #1

Hi all,
I am working on an image classification (classify scenes/characters from various animated movies) for fun during the break and realize how important it is to understand and interpret what your model learns so you can make improvement on the model or on the dataset you have (so far it helps me build a solid validation set + fix a data collection error which could lead to data leakage). Based on Jeremy’s lesson 6 pet nb, henripal’s notebook and GradCam paper, I write some code to quickly generate gradcam and guided backprop, and would love to contribute it to fastai library.

Here are few examples
(Notation: Gradcam -> GC, Guided Backprop -> GBP
Images from left to right: original image / GC w.r.t Predicted label(with probability) / GBP w.r.t Predicted label / GC w.r.t Actual label (with probability) / GBP w.r.t Actual label)

Another example with only the gradcam w.r.t predicted label (programmable)

Here is the code / example and in short, you only need these lines of code.

# from ClassificationIntepretation object. 
interp = ClassificationInterpretation.from_learner(learn,ds_type = DatasetType.Valid)

gcam = GradCam.from_interp(learner,interp,image_idx) #image_idx from ds.valid_ds or ds.test_ds
gcam.plot() #plot both GradCam and GuidedBackprop. 
# You can also choose either one by passing parameters into plot function

This can be an addition to ClassificationInterpretation, e.g after interp.most_confused() and want to find out more about those most confused classes (example in the gist). You can also do gradcam + guided bp on test set by changing ds_type to DatasetType.Test (could be good for Kaggle competition)

You can also plot heatmap + gbp for 1 single Image object (see more in the notebook)

# from a single Image object. 
img = open_image(path);
gcam = GradCam.from_one_img(learn,img)
gcam.plot()

Here is the project that I built where I use GradCam to visualize the model and troubleshoot: https://quantran.xyz/blog/building-an-image-classification-model-from-a-to-z/

Let me know if this looks good and I will refactor it a bit and do a PR

14 Likes

Developer chat
(Matt) #2

Quan-
This looks great. Would you be able to lay out how one imports your gradcam.py script, then deploy it? I think it is a bit misleading to say “you only need these lines of code” when it relies upon some dependencies. Just trying to make this more user friendly so us newbies can quickly experiment with your fine contribution!

0 Likes

(Quan Tran) #3

Hi Matt,

For local experiment, you can download the gradcam.py file to your directory and import it using from gradcam import * and then you can follow my instruction in this post for plotting. If you want to deploy it to some server, you can take a look at my deployment repo where I deploy the app to Render using Starlette.io framework: https://github.com/anhquan0412/animation-classification-deployment/tree/master/app (following this tutorial: https://towardsdatascience.com/building-web-app-for-computer-vision-model-deploying-to-production-in-10-minutes-a-detailed-ec6ac52ec7e4)

I used a simplified version of gradcam.py for deployment. This file and fastai library are the 2 deep learning library dependencies for plotting the heatmap (in server.py file)

0 Likes

(Daniel Lee) #4

Hey @quan.tran,

The code you have is awesome. It’s really helping me out. Thanks!

1 Like

(Quan Tran) #5

No problem. Glad to help!

0 Likes

(Brian Park) #6

When I try and execute from gradcam import * from gradcam-usecase.ipynb, I receive an error NameError: name 'DatasetType' is not defined from lines 5 and 7 of gradcam.py. Any thoughts?

0 Likes

(Brian Park) #7

Which fastai version are you using? Mine might be outdated.

0 Likes

(Brian Park) #8

Sorry, solved! It was my outdated library.

1 Like

(Quan Tran) #9

I was about to comment on the fastai version, but look like you already figured that out :smiley:

0 Likes

#10

Hi, I am trying to implement the GradCam function described here using fastai, but this code throws an error.
Its says, “ClassificationInterpretation object has no attribute probs”. Can some help me here. In the “gradcam.py” it seems its used as interp.probs[]. Thanks!

0 Likes

(Quan Tran) #11

Hey, sorry for the late reply. If you are using the latest version of fastai lib, the ‘probs’ attribute from ClassificationInterpretation is replaced with ‘preds’. Changing interp.probs[…] to interp.preds[…] should make the code work again (line 20). Let me know since I can’t test it right now.

1 Like

#12

Hi @quan.tran,

I got the heat maps after doing the changes you suggested. It worked as expected. Thanks for the reply.

1 Like