I am trying to train a CNN on rgb+depth data. I think it would be great to try transfer learning, to start with a resnet that has weights for rgb data, and then add a fourth depth channel on top of that. I think that the convolutions for the new channel would need to be initialized randomly, then trained.
Does this approach make any sense?
If so, does anyone have any idea how I would go about adding a new channel to the convolutions, then training it? I think that freezing generally works at the layer dimension of the network, not on the channel.