What would be the best way to choose the right pre-trained machine learning model architecture (with transfer learning) to improve accuracy in training? Currently I am just loading pre-trained weights, training for 10 epochs, then repeating this process for each pre-trained model and comparing metrics across different pre-trained models. Is there a faster way? Is 10 epochs too little/too much to train for before evaluating whether a pre-trained model is suitable to use?
And the second part to my question: Is model/arch choice in transfer learning a large factor in starting from a good base to get higher accuracy in training? Or to put it another way, should I be spending a large amount of my time choosing the correct model vs tweaking hyper-parameters?