I know that Transfer Learning is a good practice for extending existing DL architectures for most of the applications. However, there might be cases where Transfer Learning wouldn’t work. So what is the best way to create DL architectures afresh?
Is the simple approach is to keep adding layers (Convolution, Nonlinearity, Batch Norm, etc) one after the other and keep checking the accuracy? Once you think the model is doing well, then you stop.
Or there are some special considerations to make or some advanced math knowledge or beyond layperson competencies, to devise new architectures.
Any guidance will be much appreciated.