I ran an experiment and wrote a blog post looking a bit more closely at this.
There is also an accompanying repository on github with reproducible code (reproducible == open in jupyter notebook and do ‘run all cells’ 
)
I didn’t find what I was looking for which is a bit of a let down but I think there are nice ways to take the experiment further if anyone would be so inclined. This was genuinely fun to work on so I would be tempted also to look at this sometime down the road if time permits.
I also found it quite hard to do some of the things I wanted to do (for instance, looking at gradients) in keras. After just a day of playing with pytorch I can see how it make non-standard things much easier.