I find it quite often that even when the validation loss increases the accuracy increases as well.
Here is the plot:
This is a plot of a mobile net training plot. Note that even when the validation loss is around 0.3 the accuracy is around 0.9, when the loss goes to around 0.5 the accuracy is 0.94.
I can think of a couple of reasons. One is that even though the model is able to predict a few more things correctly, it is less confident about the predictions.
The plot shows the training and validation losses nicely down trending together until epoch ~30. After that their trends diverge. The validation loss then trends UP while the training loss trends down toward a limit.
It would seem that the model is overfitting sometime after epoch 20-30.
( the training is no longer helping the validation)
The accuracy peaks do drift up a few percent during while overfitting.
Maybe comparing the validation results between epoch 20 and epoch 900 would shed some light on it. Which are valid in E900 and not valid in E20? Which are valid in E20 and not valid in E900? Are any of them obivious outliers?
I should have made things clearer.
For training I am using cosine annealing rates starting with cycle len 1, cycle multiplicity 2 and 10 cycles which are basically responsible for the jumps you see. The training is done on cifar10 and the code is available here:
https://github.com/TheShadow29/pyt-mobilenet/blob/master/code/MobileNetv2-cifar-compact.py (if someone wants to take a look).
cifar10 for validation has 10k samples. 4% means around 400 images which is a pretty good margin I would say. And hence this has got me confused.
I found an interesting response by radek to a similar question.
Empirically, accuracy seems like quite a limited measure of quality of predictions. To predict whether an example belongs to some class, our model outputs a number (whatever we put through sigmoid or softmax) between 0 and 1.
To calculate accuracy, we take some arbitrary threshold (0.5 by default) and every prediction above means examples belong to some class and below they don’t. This threshold of 0.5 gets dicey really fast if we don’t have perfectly balanced class (50% positive and 50% negati…
After cycle 4 or 5 is your model getting better or worse?
Accuracy improves on the same validation set.
Cross entropy loss degrades on the same validation set.
I think you might be right that it is less confident.
Despite doing better on that validation set it starts losing generality wrt cross entropy around the end of your 5th cycle.