Weighted dataloaders with TabularPandas

fastai
1

Hi,

I am working on a healthcare problem that involves tabular dataset for training the ML model. The data has a huge class imbalance as shown below:

image
image1108×814 8.8 KB

Where the y-axis is representing the levels of cardiac rejection in transplant patients. There are more records in class 0 than in the rest. The learner is struggling to predict the rare classes as accurately as shown in the confusion matrix below:

image
image950×976 46.9 KB

Can we use weighted dataloaders like vision problems with TabularPandas? I have tried it but getting the following error:

image
image1600×1186 193 KB

Any ideas about what am I missing here?

Many thanks and
Kind regards,
Bilal

2

Not sure if I understood your code correctly or if this is at all related to the error: I think weighteddl expects weights for the training dataset only. Is your wgts list perhaps too long as it seems to be computed from the entire dataframe?

A notebook that helped me to understand the topic better is here:

3

Hi Jurgen,

Thanks for the tip.

Surprisingly, I used the same approach with vision and it worked.

I will look into this and report back.

Many thanks and

Best regards,
Bilal