ConvNets from the PDE perspective

Hi all, it’s been a long time since I last posted here (seems like another era, pre-COVID). This is an update of ConvTwist, Introducing Convolutional Layer with a Twist

Instead of “hand-engineer” the twisting, I now think it’s better to let the network learn its way. To do that, it seems that we need to use depthwise convolution, which by itself did rather well on Imagenette. (Maybe just luck on a small dataset; I don’t think anyone followed up on that.)

At any rate, I’d like to draw your attention to another post I wrote recently, on another platform, in response to ConvNext. If anything could be called the “ConvNets of the 2020s”, I’d say those that are designed from the PDE perspective. In this post, I also mentioned, in passing, that ConvNets are likely to be “hyperbolic” equations, i.e., ones that are used to model wave propagations, and how chaos and integrability of nonlinear dynamics shed lights on Deep Learning.

By happenstance, I came across a series of posts on Graph Neural Networks (which I knew nothing about) by Michael Bronstein, and his Geometric Deep Learning blueprint. It seems to me that a natural follow-up is to study and implement “hyperbolic equations on graphs”, and “hyperbolic equations on hyperbolic spaces” – there, by the way, are two distinct notions of hyperbolicity from different branches of math, see Uniform tilings in hyperbolic plane - Wikipedia. I’ll report here if anything comes out of it.

1 Like