Hi @ytian, hope you’ve been able to make some headway here! I’ve also worked on a similar problem and found some other resources that other forums members have shared. In particular, the guide mentioned in this post was extremely helpful.
I’ve created a Github gist that lays out my own steps (although with specifics redacted), and the link is at the bottom of this post. Hopefully this is useful to you!
One of the best things about the Fastai library is that the layered learning rates are absolutely crucial here. For example, you might create a model that has layer groups like:
┌───> conv1 ───> conv2 ───┐
│ │
data ┤ ├───> final ───> output
│ │
└─────────────────> tab1 ─────┘
as is the case in my example. The final + tab1 linear layers are grouped together, and so appropriately can be trained using the same learning rate. For the pretrained conv1 and conv2 layer groups, we may want to use a substantially lower learning rate. Anyway, just my two cents.