I’m trying to predict on a dataset of augmented imagenet pictures where objects are placed on different backgrounds. I am getting really low scores (pretty much random), so I’m suspecting that when I initialize the learner, the last couple of layers are re-initialized, instead of retaining the weights of the final fully-connected layers trained on the original imagenet database. Is this correct? And if so - is there a way to retain the old weights?