How can one remove the pooling from pretrained model?

Part 1 (2018)
1

I would like to use VGG16 pretrained model but replacing the maxpooling at the end of the conv layer with a Flatten so it is comparable to the keras model.

I tried xtra_cut=3 and custom_head=myhead. However this removes the last 3 conv layers and leaves the maxpool layers in place.

Is it possible to load the pretrained weights and then remove some layers?

2

Partly found the answer. The xtra_cut removes layers from the first block. The pooling is in the second block.

Using xtra_cut=0 and custom_head fails with an index error (don’t know why).

Using xtra_cut=1 and then add back the removed layer in the custom_head it now creates the learner and looks OK…however now when I try the precompute it says “ValueError: array trailing dimensions do not match with self”.

I am sure there must be a really simple way to do this.

1 Like
3

I don’t think, there is a simple way :slight_smile:
Try this

sz=224
arch=vgg16
bs=64
nf = 512*14*14
data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
learn = ConvLearner.pretrained(
    arch, 
    data, 
    precompute=False, # Disable precompute
    xtra_cut=1, # Remove max pooling
    custom_head=nn.Sequential(nn.Linear(nf,2)) # Whatever
)
learn.models.top_model.add_module('43', Flatten())
learn.models.nf = nf 
learn.set_data(data,True) # Precompute
4

Getting closer but still doesn’t work. If I look at learn.models.top_model the Flatten is there. But if I look at learn.model it is missing. And on the set_data it fails with size mismatch.

Got it working without precompute. Can confirm that vgg16 with flatten and 256/128 dense layers is massively better than the maxpool version in fastai. I suspect it is because I have small images. I wonder if fastai could be adapted to be more flexible?

5

You can actually do this using nn.Sequential quite easily. Have a look at my notebook: https://github.com/TheShadow29/FAI-notes/blob/master/notebooks/Using-Pretrained-Pytorch-Models.ipynb.

Specifically, get the pretrained model using md = learn.model. Print the model, keep the model till before the max pool. After that add a custom head. Suppose the max pool is at layer -5, you can do new_md = nn.Sequential(*list(children(md))[:-5], custom_head). custom head basically includes everything you would need after the max pool. once you have the new model you can create a new learner using learn_new = ConvLearner.from_model_data(new_md, data).

Hope this helps

6

Got the @bny6613 solution working. Precompute only works with pretrained models so have to use convlearner.pretrained. However it has to be called with precalculate=False so we can set the nf variable manually.

The reason it did not work before was I already had a fastai vgg16 model precalculated so the cache was present but different size. Working code is:

> def vgg16_keras(pre):
>     model = vgg16(pre)
>     model = nn.Sequential(*list(children(model)), Flatten())
>     return model
> 
> head = nn.Sequential(
>         nn.Linear(512 * 1 * 28, 128),
>         nn.ReLU(True),
>         nn.BatchNorm2d(128),
>         nn.Linear(128, 256),
>         nn.ReLU(True),
>         nn.BatchNorm2d(256),
>         nn.Linear(256, 2)
>     )
> 
> arch = vgg16_keras
> learn = ConvLearner.pretrained(arch, data, precompute=False, custom_head=head, opt_fn=optim.Adam)
> learn.models.nf = 14336
> learn.set_data(data,True)
7

@simoneva You can use the precompute thing without pretrained as well, though that would need you to modify a bit of code. After you have created the learner object you could repeat the steps that you noted, and use learn.save_fc1 I believe and set learn.precompute=True