I’m wondering whether anyone here has played around with the NODE algorithm from Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data & this repo. It seems like the kind of thing that @jeremy might be particularly keen on.
I ran their examples and reproduced the results from the paper, but running on my own (admittedly small) datasets, I’m not getting anywhere near a basic random forest. Also, even on a GPU, NODE seems to be painfully slow. I suspect you might be able to speed things up considerably using high / one-cycle learning rates, but haven’t had a chance to play yet.