I find it quite often that even when the validation loss increases the accuracy increases as well.
Here is the plot: https://plot.ly/~TheShadow29/1/
This is a plot of a mobile net training plot. Note that even when the validation loss is around 0.3 the accuracy is around 0.9, when the loss goes to around 0.5 the accuracy is 0.94.
I can think of a couple of reasons. One is that even though the model is able to predict a few more things correctly, it is less confident about the predictions.
The plot shows the training and validation losses nicely down trending together until epoch ~30. After that their trends diverge. The validation loss then trends UP while the training loss trends down toward a limit.
It would seem that the model is overfitting sometime after epoch 20-30.
( the training is no longer helping the validation)
The accuracy peaks do drift up a few percent during while overfitting.
Maybe comparing the validation results between epoch 20 and epoch 900 would shed some light on it. Which are valid in E900 and not valid in E20? Which are valid in E20 and not valid in E900? Are any of them obivious outliers?
I should have made things clearer.
For training I am using cosine annealing rates starting with cycle len 1, cycle multiplicity 2 and 10 cycles which are basically responsible for the jumps you see. The training is done on cifar10 and the code is available here: https://github.com/TheShadow29/pyt-mobilenet/blob/master/code/MobileNetv2-cifar-compact.py (if someone wants to take a look).
cifar10 for validation has 10k samples. 4% means around 400 images which is a pretty good margin I would say. And hence this has got me confused.
I found an interesting response by radek to a similar question.
After cycle 4 or 5 is your model getting better or worse?
I think you might be right that it is less confident.
Despite doing better on that validation set it starts losing generality wrt cross entropy around the end of your 5th cycle.