For the NLP tasks, the measure used is the perplexity, which is the exponential of the validation/test loss. So lower is better and this is where amsgrad actually hurts training the most (since there is a substantial spike in ppl).
For the tests with SGD, I need 150 epochs to get to the same perplexities on wikitext-2 (I had posted a notebook about this earlier in this thread) but someone may find a best set of hyper-parameters to get as fast as AdamW.
On CIFAR-10, SGD gets to the same accuracy in 30 epochs, but needs 20 epochs to reliably get to >94% with TTA (instead of 18 for AdamW). I haven’t tried vanilla SGD on Stanford cars, so I have no comparison for this third task.