Heatmap for Image Regression

Deep Learning
1

I am using cnn_learner() for an image regression task for a problem similar to the Biwi Image Regression Example here - https://docs.fast.ai/tutorial.data.html.

The model prediction is a point in the image where it thinks a particular object is located. This is working as expected with a red dot showing on the image where it think the object is most likely to be. Is it possible to also see a heatmap, of other likely places we may find the object?

This would be useful for my task as my images will come as stills captured from video footage and it would be useful to layer heatmap data from previous frames to more accurately track an object through time.

1 Like
2

Maybe it has something to do with running the net backwards? You feed in different x/y cords into the output and see what neurons it activates at the input? I honestly don’t know, but I really want to learn because I could learn a lot.

3

@liammmm I too am recently interested in heatmaps. It seems that the cnn_learner includes a plot_top_losses() for which you can set heatmap = true. See https://docs.fast.ai/vision.learner.html#_cl_int_plot_top_losses

Here is a tutorial that talks about this, although it doesn’t really get to the heatmaps until the very end: https://heartbeat.fritz.ai/heatmaps-and-convolutional-neural-networks-using-fast-ai-16d5b7d02a86
And here’s an earlier thread about plot_top_losses: Heatmaps in plot_top_losses()

Apart from the plot_top_losses, one can define a hook into the model and use that to get information usable for a heatmap. Here are some notes taken from an earlier iteration of Fastai where the hooks are used for heatmaps: https://medium.com/@lankinen/fast-ai-lesson-6-notes-part-1-v3-646edf916c04

Probably there are even better, more current responses to this that others can supply, but this is what I know right now.

3 Likes