How to speed up model evaluation for a large model?

I have a model with a loss curve that drops from 3m to 8000 in first epoch; then flat for 50 epochs; then a steady drop from epoch 50 to epoch 100 before flattening out. So 10 hours pass before any evidence that the model is working. I want to try different models but:

  • how do I test models fast to identify the best ones for “full” training?
  • should I avoid 1cycle learning as I don’t know the number of epochs required?