Hi,
I am continuing to explore Siamese network in fastai2. I am trying to ‘hook’ in GradCAM for a Siamese model, at the ‘encoder’ layer, but I am not sure how to extract the activations output and gradient for both ‘passes’ of the two images that form the Siamese image pair…?
From the notebook, as per definition the Siamese model’s forward pass calls the same ‘encoder’ twice, on the two images, and concatenate their output together before calling ‘head’.
I have defined my hooks:
class Hook():
def __init__(self, m):
self.hook = m.register_forward_hook(self.hook_func)
def hook_func(self, m, i, o): self.stored = o.detach().clone()
def __enter__(self, *args): return self
def __exit__(self, *args): self.hook.remove()
class HookBwd():
def __init__(self, m):
self.hook = m.register_backward_hook(self.hook_func)
def hook_func(self, m, gi, go): self.stored = go[0].detach().clone()
def __enter__(self, *args): return self
def __exit__(self, *args): self.hook.remove()
And I created a test image-pair, and applied the transforms using the defined dataloaders:
img1 = PILImage.create(Path('/path/to/image1.jpg'))
img2 = PILImage.create(Path('/path/to/image2.jpg'))
siamtest = SiameseImage(img1, img2)
tdl = learn.dls.test_dl([siamtest])
x = tdl.one_batch()
And then I called the hooks and eval the model:
cls = 1
with HookBwd(learn.model.encoder) as hookg:
with Hook(learn.model.encoder) as hook:
output = learn.model.eval()(x[0],x[1])
act = hook.stored[0].cpu()
output[0,cls].backward()
grad = hookg.stored[0].cpu()
But this will only get me 1x act for the activations and 1x grad for the gradients, presumably from the second (i.e. final) call of encoder on the second image x[1]. I am not sure how to make it output two different sets of act and grad, for the encoder pass of x[0] and x[1] respectively.
Thoughts and comments welcome. Thank you.
Regards,
Yijin