Now that are in Optimizers Hunting Season, had anyone tried Hilbert Constrained Gradient Descent? I had this paper sitting at my todo list for what feels like ages already.
The concept itself is intriguing, to say the least. Requires to change the training loop to feed a test set in the loop, and they say it is slow BUT converge fast faster (less iterations). That’s the reason why I havent tried it yet.
Paper: https://openreview.net/pdf?id=SkMwpiR9Y7
Code: https://github.com/KordingLab/hilbert-constrained-gradient-descent