Thoughts on "Regularization is all you Need" paper?

I mostly work with tabular data. Typically this means XGBoost/LightGBM/HistGradientBoostingClassifier are going to win. But in recent years it seems like that’s finally changing. It’s starting to seem like we will be using neural nets instead of gradient boosting decisions trees (GBDTs) someday soon!

Do you agree? Any tips for practitioners? Examples or links you’ve run into where deep learning outperforms GBDTs in production?

Here are some examples that I’ve stumbled on:

Take-away. Even simple neural networks can achieve competitive classification accuracies on tabular datasets when they are well regularized, using dataset-specific regularization cocktails found via standard hyperparameter optimization.

And there is also TabNet:


Although I don’t work with Tabular data I keep watching this space once in a while to understand the impact of DL. This blog post gives a chronology of DL papers for Tabular data.

His conclusion is similar to yours.