Layer normalization in fastai

Deep Learning
1

Hi everyone,

I was wondering if you think layer normalization (paper here: https://arxiv.org/abs/1607.06450) would be deserving a spot in fastai library.

The pytorch implementation, for those interested, is the following:

class LayerNorm(nn.Module):
"Construct a layernorm module"
def __init__(self, features, eps=1e-6):
    super(LayerNorm, self).__init__()
    self.a_2 = nn.Parameter(torch.ones(features))
    self.b_2 = nn.Parameter(torch.ones(features))
    self.eps = eps
    
def forward(self, x):
    mean = x.mean(-1, keepdim=True)
    std = x.std(-1, keepdim=True)
    return self.a_2 * (x - mean) / (std + self.eps) + self.b_2
1 Like