Oh also, look out for gradient clipping. I have it enabled in my sample code, but you may want to reduce it or remove it in these experiments, since it may get in the way of exploiting super-convergence.
Oh also, look out for gradient clipping. I have it enabled in my sample code, but you may want to reduce it or remove it in these experiments, since it may get in the way of exploiting super-convergence.