Instead of adding your callback to Learner … if it is simply used for training, just include it in your call(s) to fit or fit_one_cycle. As the callback is no longer associated to your Learner, they won’t interfere with your call to get_preds()
In terms of examples, you can check out one of the custom callbacks I include in my blurr library (see here). Not sure what you’re doing specifically, but this example callback I think provides a taste of what you can do and also not get in the way of the fastai bits.
Lmk how it goes.