Create Language Model for Chemical Structures

[cdparks]

Hi @marcossantana,

Yes, I am now finding that as well. It appears that FastAI doesn’t use padding when creating the batches for the initial language model training phase. For some reason, the padding embedding still chances during training though. I just posted a question about that:

Help understanding padding in language model learner

As far as I can tell, FastAI does not use padding when training language models for next token prediction. However, the embedding layer of the AWD-LSTM model takes a padding_idx as input. This row of the embedding should never be touched, as there are no padding tokens in any of the batches. I wanted to check this, so I created this toy example: > tok = Tokenizer(partial(MolTokenizer), pre_rules=[], post_rules=[]) > path = './databunch/' > > bs=64; wd=1e-2; drops=0.0 > train_su…

Where handling padding well really matters is during the classification/regression phase, as the model needs to mask the padding token when featurizing the text. This happens in the masked_concat_pool function. I was getting really bad metrics when trying to use my pre-trained LSTM to regress pIC50s. I am currently trying to figure out if there was some issue with the padding token.