I need to add a caution here. Yes, it all just works in PyTorch but maybe not in fastai v2.
The potential problem is that when fastai trains, it uses the layers specified in layer_groups to tell the optimizer which parameters to optimize. layer_groups is set when the Learner is created. If you later add new weights to the model, the optimizer will not know about them. So you may need to update layer_groups whenever you alter a model already inside a learner.
I am not sure this problem is real, and have not had the time to trace the relevant code.
For background, please see
https://forums.fast.ai/t/replicating-fastai-results-manually/61209