I tried using the image model to identify the artist of a painting. My working hypothesis is that you should be able to identify artists by various visual idioms in their paintings. So rather than classify entire paintings, I chopped up high resolution images of the paintings into small blocks and attempted to classify the artist from just the small block. As a trial run, I choose 6 artists - 3 pairs, each pair of similar style (to make it a hard problem). The artists where Botticelli, DaVinci, Monet, Renoir, Demuth, and Lewandowski (renaissance, impressionist, precisionist).
I guess I was expecting too much - since the best I could do was about 26% error rate (128x128 blocks, using resnet18, the larger resnets did slightly worse). So what can you do with the trained model? To analyze a painting, I stepped through it using overlapping blocks and got the probabilities of each artist for that block position. Then I averaged all these probabilities (per pixel) and multiplied, pixel by pixel, the original image by each artist’s probability, The resulting images show the regions of the painting which it attributes to the given artist. It’s not going to change any art critic’s mind with the results, but they are fun to look at.
Below is a example. Using a contested DaVinci painting, what does my model think?
It turns out the model is good at identifying DaVinci hands, so it got that. Interestingly, it thinks that the gold pattern on the robe is more like Botticelli. And the face is assigned to Renoir. Note that Demuth and Lewandowski images were black (zero probability), so I didn’t include them here.
I think I can do better with the following ideas that I want to try out. If I use sigmoid for the final activations (similar to object detection), then the model can indicate “don’t know”. Also, I want to included different scalings (at least 3 octaves total) to generate the training/validation image blocks. Then the analysis phase would step through the image multiple times each with a different scale size.
