I’ve looked through the first part of the course. The course is really interesting and it has good balance between practical and theoretical knowledge. Thanks a lot Jeremy, Rachel and the Fastai Team!
My question is inspired by demonstration using of CAM algorithm in the lecture 7. So, if I understand right, currently the CNN calculates their weights only for independent color channels (R, G, B). What if we create an additional 4th channel from RGB like luma (brightness, the Y component in https://en.wikipedia.org/wiki/YUV). Or, maybe does this approach already applied?
It is definitely possible and quite easy to implement too. If the data only has three channels (R, G, B) we use three channel input but if we have more channels in our data, then we would use that too. We don’t want to waste valuable information.
Oh! So the question is more about if there are cross color patterns that the network can’t see because the weights for the channels are calculated independently? It would have to be a very bad architecture to not exploit cross color patterns. In lesson 7 we used resnet34 which uses layers that don’t have this problem either.
Anyhow, you should still try it with brightness but I don’t think it will yield much better results because the state of the art models don’t use such extra channels either. If it does yield better results then you might have discovered something great!
Also, YUV from the link is also something you should experiment with. It would be easier to implement because that also has three channels.
I was thinking of channels such as alpha or a channel for infrared… those would provide more information overall.
The luma channel is typically derived from the RGB color model and represents the brightness of each pixel in the image. By separating the brightness information from the color information, it becomes possible to apply different processing techniques to each channel independently, allowing for greater control and precision in image manipulation. Visit Luma AI Art APK for more information.
