Hi,
I am trying to detect the presence of different types of clouds in night sky images, the cloud colour depends significantly on the Sun’s and the Moon’s position at the time of image capturing and day of the year and so on. I am wondering how I could feed in this data to a CNN along with the image.!
(As of now I have done first three topics of part 1 and looking to implement multi label image classification on this data)
This is one of ex. image
Using Metadata together with raw image data in classification sounds like a really cool idea.
But first: Do you really need this data for better cloud identification? If you have enough images, wouldn’t the CNN learn either way to identify the correct cloud? Or maybe you can pre-process images by applying proper white-balancing, so that clouds are “white” (or gray scale) either way?
But let’s say we want to go with the metadata route. The CNN part itself is really meant for image processing, and I’m assuming you’ll be, for the most part, finetuning a resnet or something. So I feel like “naively” concatenating the metadata with the image data would give poor results. Instead, I’d process the image part with a CNN-based architecture and the metadata part using the techniques for structural data, and then at some point splice together the outputs from the two NNs into some final layers.
@cadolphs Thank you very much for your response.
I feel that the information in metadata would be very useful for the classification. As far as I understood, CNN is looking for structures in the images. The cloud on other hand could take many different shapes. One of the important and reliable feature of cloud is its colour. Background sky could some time look like cloudy white or complete smooth cloud cover could some time look like blue clear sky which depends on the Sun and Moon position in the sky, season, exposure time, ISO etc.
Your idea to merge two networks is nice 
I am wondering whether the network would learn to give significant importance to metadata as the number of pixels in an image is far greater than just five to six values of metadata? Anyways, I will start with it and see what the network is doing. If you are interested, please let me know I will give you an update about what the network is doing.
Thank you for your time.
Hello @cadolphs, I developed a simple binary classification model just to detect clear sky or cloud (not different type of clouds, just to start with it). I saved the model and I want to run on another system which does not have gpu. I made another post (How to load fastai saved model with Pytorch-cpu). Could you please look at it and provide any suggestions.
Thank you.
Sorry, I can’t help with that because I haven’t done something like that. I can only suggest to look closer into the error message, the documentation for the various modules you are using and maybe use google to see if others have gotten a similar error message.
Yeah, I checked for that… Many have the same problem. But, no solution. Let’s see… anyways thank you for your time.
