Here’s some style transfer results from playing around today, and accompanying thoughts for anyone interested.
I thought it would be interesting to see if style transfer might work on extracting style from something like architectural design. St. Basil’s Cathedral in Moscow seemed a good choice for this sort of thing, given it’s color scheme and style. I arbitrarily selected the White House as the content image.
With content output at block3conv2:
This simultaneously looks like a pretty pastel and a foreboding image, what with the red sky. The White House is still completely discernible, as well as the flag and lawn. The bushes on the left and right are quite “blobby”; it’s unclear whether we would know they were bushes if we didn’t already know that beforehand.
With content output at block4conv2
Still discernibly the white house, but content is looser here at a later layer. Slightly nightmarish, the skyline looks positively apocalyptic…
Neither of these gave any indication that the style transfer was incorporating any of the geometric color patterns, like the swirls on the domes. Here’s the deconstructed style:
Has all the colors, and some of the “stripey-ness” of the domes. But not enough to really transfer those striping patterns. If I had to guess, I’d say it’s probably because the domes are small parts of the image.
I played around with a lot of different loss weighting/ content conv output combos for different images, and usually the best ones were block3conv2 with scaling 1/10 on content loss. Block4conv2 with no content scaling usually gave good results too, but much more “trippy”. The rest of the images have been done with the first scheme.
Here’s another one from a different White House image and a different image of St. Basil’s:
Equally spooky.
Moving on to style from art. In Jeremy’s examples during lecture, the one example that didn’t seem to work well was the Simpson’s one. This has been mentioned already, but I’m fairly certain the reason that particular style didn’t work so well was because with a cartoon, style transfer isn’t doing what we expect it should. When we think of drawing a bird in the style of the Simpson’s, that means completely re-drawing the bird’s edges and changing the content completely to look like a Simpson’s cartoon. The only “style” that defines the Simpson’s is their edges; everything else is flat color. Style transfer won’t do that with this kind of cartoon.
When I tried with a more textured cartoon:
And applied it to POTUS:
The result reflects the original cartoon style much better.
Specifically, notice that The Donald has the same white/black silhouette as Vlad. The folding in the flag and clothes are much more pronounced than in the original image, matching those in the cartoon. It really has accentuated the almost unnoticeable edges in the content image, turning them into hard lines.
The best looking results I got though were predictably those from more impressionist images like Picasso:
Applied to Saint Pablo himself:
Results:
But I think what truly astonished me was how well style transfer worked on things that weren’t necessarily in the foreground.
I applied style transfer on a frame from the Love Lockdown music video:
And the result:
That really blew my mind. You can absolutely tell that is a man sitting in that corner, with what looks like less than ten brush strokes. The only clarity that’s lost here is the corner between walls he’s leaning in, which looking at the original image is almost impossible to see; and the finer details of the individual on the right. Everything else, the wall-floor edges, the framing, that’s all preserved.
I’m very interested in learning how to do this for a video. As of yet, there doesn’t seem to be an open source python implementation based off this paper https://arxiv.org/abs/1604.08610 , although there is one in lua.
