Indexing into model[0] (= timm body) when using timm library with fastai

hey, so I wanted to use gradcam via a hook with a timm model trained with fastai. (I used vision learner and everything from fastai, except the timm model).

Now I can’t index into the layers of the model body (equal to model[0]) when it is a timm model and I get the error message that I can’t index into a “timm body”.

Are there other ways to index into the model body of a timm model for feature extraction or grad cams etc?

I switched now to a model directly from fastai to be able to use gradcam (via a hook).

Best, Sophia

1 Like

Can you post some code or a notebook that will replicate the issue? Its tricky to advise how to change your code without seeing what is written, although I’m speaking for myself as there are some legends here that can!

From the docs maybe you index with smooth brackets ‘model(x)’ rather than square? ‘model[x]’? As shown in the Fastai Create Timm Model Docs docs. Again not sure if this is relevant to your situation since I can’t see a code stub.

I’m guessing from a distance though, a colab I could run and debug would be awesome, hope that helps, otherwise if you post back a stub I’d be stoked to work through it with you :blush:

1 Like

Hey, thank you for your reply. :grin:I didn’t have the code at hand anymore, but you re totally right about adding code, below is an example. Just use any model written with " " around the name (therefore the model is then used from the timm library) and you have the problem as shown below. I “solved” it by using grad cam instead of using cam (class activation maps) for the wanted location of the output, but I still cannot use grad cam this way for “any” location within the body of the model (within the timm body).

Example from the fastbook I used for grad cam:

class Hook():
    def __init__(self, m):
        self.hook = m.register_forward_hook(self.hook_func)   
    def hook_func(self, m, i, o): self.stored = o.detach().clone()
    def __enter__(self, *args): return self
    def __exit__(self, *args): self.hook.remove()
        
        
class HookBwd():
    def __init__(self, m):
        self.hook = m.register_backward_hook(self.hook_func)   
    def hook_func(self, m, gi, go): self.stored = go[0].detach().clone()
    def __enter__(self, *args): return self
    def __exit__(self, *args): self.hook.remove()

for index, i in enumerate(range(len(dls.valid_ds))):
    x, = first(dls.test_dl([dls.valid_ds[i][0]]))
cls = 1
with HookBwd(learn.model[0]) as hookg:
    with Hook(learn.model[0]) as hook:
        output = learn.model.eval()(x.cuda())
        act = hook.stored
    output[0,cls].backward()
    grad = hookg.stored
    
w = grad[0].mean(dim=[1,2], keepdim=True)
cam_map = (w * act[0]).sum(0)


x_dec = TensorImage(dls.valid.decode((x,))[0][0])

_,ax = plt.subplots()
x_dec.show(ctx=ax)
ax.imshow(cam_map.detach().cpu(), alpha=0.6, extent=(0,224,224,0),
              interpolation='bilinear', cmap='magma');


_,ax = plt.subplots()
x_dec.show(ctx=ax)
ax.imshow(cam_map.detach().cpu(), alpha=0.6, extent=(0,224,224,0),
              interpolation='bilinear', cmap='magma');

code example that creates the mentioned problem:

path = untar_data(URLs.PETS)/'images'
def is_cat(x): return x[0].isupper()
dls = ImageDataLoaders.from_name_func(
    path, get_image_files(path), valid_pct=0.2, seed=21,
    label_func=is_cat, item_tfms=Resize(256))
learn = vision_learner(dls, "convnext_nano.in12k_ft_in1k", metrics=error_rate)
learn.fine_tune(0)
img =PILImage.create(get_image_files('/media/testset_images')[0])

x, = first(dls.test_dl([img]))



hook_output = Hook()

hook = learn.model[0][5].register_forward_hook(hook_output.hook_func)

with torch.no_grad(): output = learn.model.eval()(x)

act = hook_output.stored[0]

act.shape

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In [8], line 6
      2 x, = first(dls.test_dl([img]))
      5 hook_output = Hook()
----> 6 hook = learn.model[0][5].register_forward_hook(hook_output.hook_func)
      7 with torch.no_grad(): output = learn.model.eval()(x)
      8 act = hook_output.stored[0]

TypeError: 'TimmBody' object is not subscriptable
1 Like

Hey,
it seems that indexing into the model only works for sequential models. You can navigate to particular layers with the layers names (combined with the indices if the parts of the network are wrapped in a nn.Sequential). You can get the models summary with learn.model and read of the layer names:

Hope that helps.

2 Likes

Hey @sophia sorry I didn’t get back to you earlier, totally spaced on coming back to the forum.

Thanks @benkarr for providing a good explanation :smiley: , I didn’t know you could traverse down the model that way.

I also like the get_submodule() method from lesson 3 from pytorch’s nn module because you can not worry about the indexing & attributes and instead just have a dot separated string to get to your layer

And then you can get this layer specifically via this string dot notation:

2 Likes