Training Fastai pretrained models

Deep Learning
#1

Hi all - just a general curiosity question to check my understanding of things.

For Fastai specific models such as the unets (https://docs.fast.ai/vision.models.unet.html) is there any documentation on how often these are trained; and what the process is when training them from scratch to create the pre-trained baselines?

Thanks

1 Like
#2

My understanding is only the backbone is pretrained. And most of the supported backbones comes from torchvision libraries.

#3

Very interesting; as mentioned; this is very much an exploratory discussion from my side! All pointers and thoughts much appreciated.

Poking around in the code I’ve come across -

class DynamicUnet(SequentialEx):
    "Create a U-Net from a given architecture."
   ....

https://github.com/fastai/fastai/blob/master/nbs/15a_vision.models.unet.ipynb

This model seems to have quite a bit of “bespoke” logic; but could weights trained with some other unet model be “re-used” and still give accurate results?

I don’t actually see a “unet” model in https://pytorch.org/docs/stable/torchvision/models.html … maybe they have a different naming there?

I think the a reasonable point of reference for the fastai ResNet model can be looked at here

class ResBlock(Module):
    "Resnet block from `ni` to `nh` with `stride`"

https://github.com/fastai/fastai/blob/master/nbs/01_layers.ipynb

And again; it seems quite bespoke. My guess was it would require its own training cycle; but if its actually possible to re-use waits from “similar” pytorch models; that would be … good to know, and highlight I’ve got a lot more understanding to get my head around!

Are those reasonable places to consider the definition of fastai unets and resnets?

#4

With all these models only the backbone is pretrained, even with the dynamic unet. The rest of the layers are just initialized. And these pretrained models gather their weights from the torchvision models.

Now you can use any pretrained model you want and apply transfer learning, as discussed in the course.

fastai models are just PyTorch models. So as long as you use the same model, you can use the same weights. I have a lesson on this from my course here you may be interested in @bgraysea

1 Like
#5

{Exploding brain}

I’m not sure I understand fastai models are just PyTorch models. — is the fastai code I linked in my 2nd post not part of the backbone at all?

Could you link me to where fastai “starts using the pytorch backbone models” in the fastai library? Maybe that’ll help me get the correct idea of how the fastai bespoke layers interact with the backbone.

Thanks for the link; I’ll go through that as it seems like it will be very helpful.

#6

Nope it’s not. That’s the top part of the unet model. fastai is built on PyTorch, hence why all fastai models must be PyTorch models. Now where these pretrained models come from, you should look at the source code for cnn_learner and unet_learner. You’ll find some code called create_body, which grabs a PyTorch pretrained model (when we pass say resnet34), and cuts off the last layer (that linear layer)

1 Like
#7

Now where the PyTorch models come from, they’re imported here: https://github.com/fastai/fastai/blob/master/fastai/vision/models/tvm.py

And made into a namespace here with all the information cnn_learner needs :

We’re also looking at fully integrating the timm library now instead, so this will change in fastai 2.3+

#8

Right; I can pretty much follow along with that; I’ll definitely have to re-look at the code and the lessons in light of these understanding corrections; hugely appreciated; thanks

1 Like