I’d like to try out Hebbian Softmax training (https://arxiv.org/abs/1803.10049) which improves LM results, especially on infrequent words.
The algorithm would fit naturally as
on_loss_begin callback because it requires the outputs of the RNN before the LM decoder module. That’s returned by the LM (
outputs in https://github.com/fastai/fastai/blob/master/fastai/text/models.py#L150), however the
RNNTrainer callback removes it (https://github.com/fastai/fastai/blob/master/fastai/callbacks/rnn.py#L17) and that callback is hardcoded to go first in the list of callbacks.
There are multiple non-clean ways to get those
outputs to another callback but I was curious if people here had any advice on what would be most natural way to do it given the design of the library and any forthcoming changes.