As I’m wanting to try out various new tabular models, I keep drawing to the same 2 datasets we use without much variation (ADULTs and Rossmann), so I’ve constructed a baseline (of sorts) in which
fastai can be compared with. The repository is here including a notebook showing how I achieved these baselines here. The baselines themselves were taking from the TabNet paper, which was published in September of 2019. The goal is to present three things: the model, it’s accuracy, and the number of total parameters in the model. Also, as a request, if you do work with these try to post your feature importance as well, as there could be very interesting developments to where everyone’s model leans
Onto the baselines:
The three proposed datasets are:
Challenge: Successfully identify the rank of the current hand based on suit (categorical) and rank (numerical) for the five cards in your hand (total of 10 variables).
Model Test Accuracy (%) Decision Tree 50% Multi-layer perceptron 50% Deep Neural Decision Tree 65.1% TabNet 99.10% fastai2 99.48%
Challenge: Map 21D space components to successfully estimate the torque.
- Rankings: (fastai won this one!)
Model Test MSE Number of Parameters Random Forest 2.39 16.7K Stochastic Decision Tree 2.11 28K Multi-Layer Perceptron 2.13 0.14M Adaptive Neural Tree Ensemble 1.23 0.60M Gradient Boosted Tree 1.44 0.99M TabNet-S 1.25 6.3K TabNet-M 0.28 590K TabNet-L 0.14 1.75M fastai2 0.038 530K
- Challenge: Using simulated data with features characterizing events detected by ATLAS, classify events into “tau tau decay of a Higgs boson” versus “background.”
Model Test Accuracy (%) Number of Parameters Sparse evolutionary trained multi-layer perceptron 78.47 81K Gradient boosted tree - S 74.22 120K Gradient boosted tree - M 75.97 690K Multi-layer perceptron 78.44 2.04M Gradient boosted tree - L 76.98 6.96M TabNet - S 78.25 81K TabNet - M 78.84 660K fastai2 76.94 530K
Hopefully this can help some of you keep track of reasonable goals with tabular models (and hopefully some of you beat these!)