Thanks, @pete88b. As I mentioned, there’s a lot of waste in parameters.
If you are testing on ResNeXt, did you give each conv2d groups argument? (That’s as much as I understand ResNeXt)
Briefly, I’m adding two extra conv2d (that I call convx and convy), but you can see that I “symmetrized” the weights, so instead of 9 parameters for each filter/channel/feature-map, it’s only 4 in effect. I also copied the convx weights into convy so the entire convy is extraneous. Of course once we have tested various possibilities we could write the TwistLayer more efficiently.
You asked where you can learn about TwistLayer. It’s related to the Neural ODE paper (and others) that interprets ResNet as solving a differential equation. I wrote about the mathematics here
but at the time I didn’t actually know ResNet (even now I know very little beyond ResNet) and I should do a complete rewrite.
I can open up a separate thread to answer questions. [Update: new thread here]