I am having problems interpreting my training results. I use the default flattened MSE Loss to do regression on a dataset, and RMSE is used as ``metrics’’ when creating the Learner.
However, the problem is, the RMSE is not equal to the root of the valid loss (which is MSE). It is always smaller.
epoch
train_loss
valid_loss
root_mean_squared_error
time
0
0.223325
0.192066
0.437771
00:08
1
0.170728
0.126881
0.355616
00:09
The full code:
I am using FastAI 1.0.60.
Any idea why this happens?
I love these curiosity questions about an anomaly. Sometimes they lead to great discoveries. Sometimes “only” to better understanding.
First, I do not see this issue with fastai2. I am not using fastai1 currently.
If I had to risk a wild guess about fastai1:
valid_loss: the mean of all the errors squared for the whole validation set
root_mean_squared_error: the mean of the square roots of MSE across a smaller group, like batches.
And mean(sqrt()) != sqrt(mean())
You could find out exactly what is happening by putting a debugger on the metrics. Please let us know if you find the right answer!
I find that FastAI 1 calculates the valid loss and RMSE using the same method. It first calculates the score on mini-batches, then doing a weighted average to get the final loss/score, and the weight is the number of data in the mini-batch.