I have a regression problem with a tabular data. Hence, the loss function which I am using is MSELossFlat and the metric is mean_squared_error. These should obviously give the same result on the validation set. However, it is not the case. The output is:
|epoch|train_loss|valid_loss|mean_squared_error| |1| 0.047028| 0.008886| 2.268168| |2| 0.041831| 0.034416| 2.090547| |3| 0.030867| 0.007301| 2.322598| |4| 0.038007| 0.012626| 2.236360| |5| 0.033619| 0.010785| 2.593846|
So, valid_loss and the mean_squared_error metric on the validation set are very different. Would would be the reason for that? Can it be bug.
This experiment s easily reproducible by running the tabular.ipynb in the example folder and modifying 2 cells. The third cell I modify as:
from sklearn import preprocessing #df['salary'].unique() df['salary'] = preprocessing.StandardScaler().fit_transform( (df['age'] + df['fnlwgt']).values.reshape(-1,1) )
In the cell with the learner definition I pass the loss and the metric explicitely, also change the number pf epochs:
from fastai.layers import MSELossFlat from fastai.metrics import mean_squared_error learn = tabular_learner(data, layers=[200,100], loss_func=MSELossFlat(), metrics=mean_squared_error) learn.fit(5, 1e-2)
Do you have any thoughts on this behaviour?