I just finished this super interesting paper that implies (at least to my reading) that the structure of the network is actually much more important than the training, which has really interesting implications for transfer learning.
In it they take the VGG19 network, randomly generate weights and are able to reproduce original images and do style transfer on a similar level to what we’ve been doing.
Intuitively it makes some sense after reading it. The random mappings of the neural net create an incredibly rich complex multidimensional space and when you’re trying to minimize the distance across that space by creating an image that maps to a similar activation you’re going to come up with similar images.
Still it does run counter to a lot of what we’ve been talking about and I’d be curious to hear others takes on it.