Using an additional channel for images: luma (brightness)

I’ve looked through the first part of the course. The course is really interesting and it has good balance between practical and theoretical knowledge. Thanks a lot Jeremy, Rachel and the Fastai Team!

My question is inspired by demonstration using of CAM algorithm in the lecture 7. So, if I understand right, currently the CNN calculates their weights only for independent color channels (R, G, B). What if we create an additional 4th channel from RGB like luma (brightness, the Y component in Or, maybe does this approach already applied?

It is definitely possible and quite easy to implement too. If the data only has three channels (R, G, B) we use three channel input but if we have more channels in our data, then we would use that too. We don’t want to waste valuable information.

Maybe you know, will the accuracy of a model increase the addition of a new channel which calculated from existing channels?

Oh! So the question is more about if there are cross color patterns that the network can’t see because the weights for the channels are calculated independently? It would have to be a very bad architecture to not exploit cross color patterns. In lesson 7 we used resnet34 which uses layers that don’t have this problem either.

Anyhow, you should still try it with brightness but I don’t think it will yield much better results because the state of the art models don’t use such extra channels either. If it does yield better results then you might have discovered something great! :slight_smile:

Also, YUV from the link is also something you should experiment with. It would be easier to implement because that also has three channels.

I was thinking of channels such as alpha or a channel for infrared… those would provide more information overall.