PyTorch - Best way to get at intermediate layers in VGG and ResNet?

(WG) #1

Just getting started with transfer learning in PyTorch and was wondering …

What is the recommended way(s) to grab output at intermediate layers (not just the last layer)?

In particular, how should one pre-compute the convolutional output for VGG16 … or get the output of ResNet50 BEFORE the global average pooling layer?

Getting activations of certain layer after model is trained
(WG) #2

Here is at least one way to do it … thoughts?

res50_model = models.resnet50(pretrained=True)
res50_conv = nn.Sequential(*list(res50_model.children())[:-2])

This grabs a pretrained resnet50 model courtesy of the torchvision package and then builds a sequential model based on it that excludes the final two modules (e.g., the one that does average pooling and the fully connected one)

for param in res50_conv.parameters():
    param.requires_grad = False

No need to backprop through the model since I’m using it purely for feature extraction.

inputs, labels = next(iter(dataloders['train']))
inputs, labels = Variable(inputs), Variable(labels)
outputs = res50_conv(inputs)

To test, I grab 4 examples and run them through my modified model. # => torch.Size([4, 2048, 7, 7])

And voila, I get the 2048x7x7 output I expected!

It’s feels both weird and cool to be able to pass in images of any size into the network and that it just works. I burnt a few minutes here and there trying to get the model to tell me the output size of this layer or that layer until I realized it only works for the fully connected layers because, I believe, those are the only ones that do have a definitive input and output shape. The convolutional layers inputs/outputs shape will b dynamic based on the shape of your examples … which like I said, feels both weird coming from using Theano/TF but also very cool.

(WG) #3

Here is another way inspired from this forum post:

class ResNet50Bottom(nn.Module):
    def __init__(self, original_model):
        super(ResNet50Bottom, self).__init__()
        self.features = nn.Sequential(*list(original_model.children())[:-2])
    def forward(self, x):
        x = self.features(x)
        return x

res50_model = models.resnet50(pretrained=True)
res50_conv2 = ResNet50Bottom(res50_model)

outputs = res50_conv2(inputs)  # => torch.Size([4, 2048, 7, 7])

Also, forgot to mention that I’m tweaking things from the transfer learning tutorial on the pytorch website. Check it out to understand the dataset and other particulars.

(Krishna Vishal V ) #6

To get output of any layer while doing a single forward pass, you can use register_forward_hook.

outputs= []
def hook(module, input, output):

res50_model = models.resnet50(pretrained=True)
out = res50_model(res)
out = res50_model(res1)

Here in the outputs you get a list with two tensors in it. Those tensors are outputs of that particular layer for each forward pass.

(D.X) #7

I tried to change the first maxpool layer to avg_pool layer in resnet34, I did this but only got a maxpool missed model, could you tell me why doesn’t it work, how to do that?

class resnet30_avg(nn.Module):
def init(self, original_model):
super(resnet30_avg, self).init()
self.S1 = nn.Sequential(*list(original_model.children())[:3])
self.S2 = nn.Sequential(*list(original_model.children())[4:])

def forward(self, x):
    x = F.avg_pool2d(self.S1(x), 3, stride=2, padding=1)
    x = self.S2(x)
    return x


(S1): Sequential(
(0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace)
(S2): Sequential(
(0): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)