Is there an easy way to check which layers in your model are frozen?
I know that learner.freeze_to(0) unfreezes all the layers, and learner.freeze_to(-1) unfreezes all the layers except for the last.
But what if I just want to check which layers are frozen?
Thanks
You can check the layer’s trainable attribute.
From elsewhere in the forums it looks like the way you can check out the layers is through model.layers.
For an RNN model, it doesn’t look like that’s the case:
AttributeError: ‘RNN_Learner’ object has no attribute ‘layers’
Any insight on how to assess the trainable status for layers of an RNN_Learner object?
All learners have a ‘model’ attribute that contains the pytorch model. That’s a regular nn.Module - if you’re not familiar with how to navigate that, then you’ll want to check out the pytorch tutorials on modules.
@jeremy, thanks for pointing me in the right direction!
Let’s see if I got this right…
RNN_Learner has 2 layers groups: 1) the RNN_Encoder (accessible through learner.model[0]), and 2) the LinearDecoder (accessible through learner.model[1]).
The RNN_Encoder contains multiple layers and layer groups, including 1) an encoder Embedding layer, 2) an encoder dropout layer, 3) an rnns LSTM layer group, 4) a dropouti layer, 5) a dropouths layer group.
The LinearDecoder contains two layers: 1) a decoder Linear layer, and 2) a dropout layer.
The trainable status of most of the above can be accessed through:
print(learner.model[0].rnns[0].trainable)
print(learner.model[0].dropouths[0].trainable)
print(learner.model[0].rnns[1].trainable)
print(learner.model[0].dropouths[1].trainable)
print(learner.model[0].rnns[2].trainable)
print(learner.model[0].dropouths[2].trainable)
print(learner.model[0].dropouti.trainable)
print(learner.model[1].decoder.trainable)
print(learner.model[1].dropout.trainable)
learner.freeze_to(0) freezes all the above.
learner.freeze_to(-1) unfreezes [0].dropouti, [1].decoder, and [1].dropout
learner.freeze_to(-2) additionally unfreezes .rnns[2] and .dropouths[2]
learner.freeze_to(-3) additionally unfreezes .rnns[1] and .dropouths[1]
learner.freeze_to(-4) additionally unfreezes .rnns[0] and .dropouths[0]
I couldn’t find the trainable status of the encoder Embedding layer, and the encoder dropout layer… Is there some conceptual reason why there’s no learner.model[0].encoder.trainable?
Also, I was surprised that learner.freeze_to(-1) unfreezed [0].dropouti… Can you point me in the right direction to understand this?
You’ve made great progress there! Let me try to fix some of the terminology:
The RNN_Encoder is an nn.Sequential model (subclass) that contains two layers. For things that aren’t sequential models it starts to become a bit difficult to say how many layers they really have - since modules contain modules, which can contain modules, etc. And in forward the may call functions that aren’t actually parameters. Or there may be parameters that aren’t modules. (See if you can think of examples of each of these.)
Layer groups are specific to fastai, and the way to see them is learn.get_layer_groups(). They are not necessarily in the same order as the layers you see listed in the model. I’ve ordered them according to what I think is the most useful order to unfreeze them. Since I figure you’re most likely in a language model to want to fine-tune the embeddings first, that’s the last layer group (so the one that’s unfrozen first by default).
The trainable attribute is specific to fastai. To really see what will be trained you need to look at the module’s parameters attribute, and look for which have requires_grad. trainable should generally be the same as that, but if you do some stuff outside of fastai it won’t be.
Hopefully that helps deepen your understanding a little, or at least gives you ideas of what to look at next.