What does emb_init do?

edwincv0 · May 13, 2018, 6:48pm

Just had been working through the FastAI code using columnar data? Just wondering what is the purpose of this emb_init?

fastai/fastai/blob/master/fastai/column_data.py#L82


    ((val_df, trn_df), (val_y, trn_y)) = split_by_idx(val_idxs, df, y)
    return cls.from_data_frames(path, trn_df, val_df, trn_y, val_y, cat_flds, bs, is_reg, is_multi, test_df=test_df)


def get_learner(self, emb_szs, n_cont, emb_drop, out_sz, szs, drops,
                y_range=None, use_bn=False, **kwargs):
    model = MixedInputModel(emb_szs, n_cont, emb_drop, out_sz, szs, drops, y_range, use_bn, self.is_reg, self.is_multi)
    return StructuredLearner(self, StructuredModel(to_gpu(model)), opt_fn=optim.Adam, **kwargs)




def emb_init(x):
x = x.weight.data
sc = 2/(x.size(1)+1)
x.uniform_(-sc,sc)




class MixedInputModel(nn.Module):
def __init__(self, emb_szs, n_cont, emb_drop, out_sz, szs, drops,
             y_range=None, use_bn=False, is_reg=True, is_multi=False):
    super().__init__()
    self.embs = nn.ModuleList([nn.Embedding(c, s) for c,s in emb_szs])
    for emb in self.embs: emb_init(emb)

Anyways, love the library, really awesome stuff.

Edwin

bny6613 · May 13, 2018, 8:36pm

Hi. I think Jeremy covered this in Lesson 5 https://youtu.be/J99NV9Cr75I?t=46m2s

Hadus · May 13, 2018, 8:43pm

We do the first line just for convenience. We can write x instead of x.weight.data.

This means the third line is replaceing the x.weight.data with x.weight.data.uniform(-sc, sc). (also there is no not inplace version of .uniform)
Which essentially means filling in the weights of the embeddings with some initial values.
These values are generated exactly as np.random.uniform even the args that we had (min, max) are the same. Look at the numpy version’s documentation to see what kind of numbers it generates.

The second line is just getting the min and the max values for the uniform function. I don’t know what the sc stands for or why it is calculated this way… maybe someone else can help you with that.

jeremy · May 13, 2018, 10:28pm

It’s just an initializer for embeddings. As a rule of thumb it seems to work pretty well in my testing. I don’t have any rigorous basis for it however.