Hello!
I can’t find answer to “what is head of neural network“?
I found different topics like “training cnn with multiple heads”, but can’t find defenition of what head is.
Is this a bunch of layers in the beginning or in the end of network?
1 Like
This is the top of a network. It is a bit arbitrary. For instance, on the bottom (where data comes in) you take convolution layers of some model, say resnet that we use in this course. If you call ConvLearner.pretrained, CovnetBuilder will build a network with appropriate head to your data (if you are working on a classification problem, it will create a head with a cross entropy loss, if you are working on a regression problem, it will create a head suited to that).
But you could build a model that has multiple heads. The model could take inputs from the base network (resnet conv layers) and feed the activations to some model, say head1 and then same data to head2. Or you could have some number of shared layers built on top of resnet and only those layers feeding to head1 and head 2. You could even have different layers feed to different heads!
There are some nuances to this (for instance, with regards to the fastai lib, ConvnetBuilder will add an AdaptivePooling layer on top of the base network if you don’t specify the custom_head argument and if you do it won’t) but this is the general picture.
7 Likes
30 seconds with explanation
Every model might be thought of as a backbone plus a head, and if you pre-train backbone and put a random head, you can fine tune it and it is a good idea.
Looks like it is applicable to all neural networks
4 Likes