Some useful functions for tabular models

[Pak (Pavel)]

@jeremyeast

Hi Pavel, thanks for these great code shares, what kind of speed increase did you get before & after ?

If you referring to this

@Pak

in fact in my setup learn.predict in for-loop for 200 rows lasts 45+ seconds, and the process of calculating error for whole dataframe consists of 10,000 rows with standard tools take less than a second

Then… predict with learn.predict(row) is very ineffective (here is another example How to predict large unseen tabular datas with trained model? ) so in this case it’s thousands of times faster, but it’s just because learn.predict wasn’t meant for this.
In fact maybe there is a more convenient way to do that, probably involving something like .add_test, and making sure results wouldn’t be shuffled, but I ended up writing my own functions, because I couldn’t make it work.

@jeremyeast

Also other question is : did you use the m.export() file as the learner file, or did you use the m.save() file for future predictions ?

I have used just
learn.save('xxx')
and
learn = learn.load('xxx');
for saving the learner between sessions

@jeremyeast

For the plot_dendogram_corr, you use the following, but I don’t see the use of get_top_corr_dict() because you do not use its return value ?

get_top_corr_dict(df[all_vars], corr_thr=0.9, corr_matr=corr_v)

That function outputs (or prints in this case) top pairs of correlation in a given dataframe.
I did not include output itself there. But it looks like this:

OrderedDict([('Store vs CompetitionDistance', 1.0),
                 ('Store vs StoreType', 0.999),
                 ('Store vs Assortment', 0.999),
                 ('Store vs PromoInterval', 0.999),
                 ('Store vs CompetitionOpenSinceYear', 0.999),
                 ('Store vs Promo2SinceYear', 0.999),
                 ('Store vs State', 0.999),
                 ('Month vs Week', 0.965),
                 ('StateHoliday vs AfterStateHoliday', 0.962)])