Modify _load_pretrained_weights for multispectral images

In the _load_pretrained_weigths method for fastai v2, there are special cases to handle weights when the number of input channels is less than 3:

    if n_in==1:
        # we take the sum =, keepdim=True)
    elif n_in==2:
        # we take first 2 channels + 50% =[:,:2] * 1.5

However, if the number of channels is greater than 3, nothing special happens for most of them and they are zeroed:

        # keep 3 channels weights and set others to null[:,:3] =[:,3:].zero_()

My intuition is that we might gain some benefit by treating the rest of the channels as if they were a grayscale image as in the 1-channel case, e.g., something akin to (conceptually - am not sure this compiles; probably need to replicate the grayscale data across the number of remaining dimensions):

        # keep 3 channels weights and set others to a grayscale equivalent[:,:3] =[:,3:] =, keepdim=True)

Has anyone attempted this? Did it help with transfer learning to multispectral data?

Actually it looks like @Nickelberry has solved this problem with a similar solution and discussed in this thread. My main question at this point is whether that ought to be the default, i.e., whether it’s worth a PR for fastai?

1 Like