Tabular - Low Train Loss and Valid Loss, High Error Rate

I am training a Tabular model but I keep getting very low Train and Valid Loss but the error rate is always = 1.0

Lr finder :
image

lr.valley = 0.0020892962347716093

learner.fit_one_cycle(2,0.002)
preds,targs = learner.get_preds()
mean_squared_error(targs, preds, squared=False)

epoch train_loss valid_loss error_rate time
0 0.082041 0.042345 1.000000 05:50
1 0.033453 0.016462 1.000000 05:30

mean_squared_error = 0.12830542

I tried to increase the numbers of rows (50k, 100k , 300k), but it doesn’t change anything. The more I train it, the better my Train and Valid Loss get but the Error rate stay the same at 1.0.

I tried with the accuracy settings:

epoch train_loss valid_loss accuracy time
0 0.144856 0.018036 0.000000 03:40
1 0.081213 0.015431 0.000000 03:38
2 0.062788 0.015088 0.000000 03:27

mean_squared_error = 0.12283298

Anybody had a similar issue ? What could be the problem ?

Hello,

Is this a regression or classification task? The accuracy and error rate are suitable only for the latter and cannot be used with continuous labels. However, it seems to me that your loss is the mean squared error, indicating that this is indeed a regression problem.

Thank you.

1 Like

I don’t have a great deal of experience with this, but notice here it specifies a particular metric to use…

learn = tabular_learner(dls, layers=[1000,500],
                        metrics=exp_rmspe,
                        config=tc,
                        loss_func=MSELossFlat())

… and there are probably others suitable:

Thank you both for your reply, I will look into it and come back with the results. It is indeed possible I’m using the wrong metric.

2 Likes