Switchnorm - possible candidate from fastai

Gabriel_Syme · July 18, 2018, 4:59am

I’m just sharing the code for a new normalization layer called switchnorm:

switchablenorms/Switchable-Normalization/blob/master/models/switchable_norm.py

import torch
import torch.nn as nn

class SwitchNorm(nn.Module):
    def __init__(self, num_features, eps=1e-5, momentum=0.997, using_moving_average=True, last_gamma=False):
        super(SwitchNorm, self).__init__()
        self.weight = nn.Parameter(torch.ones(1,num_features,1,1))
        self.bias = nn.Parameter(torch.zeros(1,num_features,1,1))
        self.mean_weight = nn.Parameter(torch.ones(3))
        self.var_weight = nn.Parameter(torch.ones(3))
        self.eps = eps
        self.momentum = momentum
        self.using_moving_average = using_moving_average
        self.last_gamma = last_gamma
        self.register_buffer('running_mean', torch.zeros(1, num_features, 1))
        self.register_buffer('running_var', torch.zeros(1, num_features, 1))
        self.reset_parameters()

    def reset_parameters(self):
        self.running_mean.zero_()

This file has been truncated. show original

It’s in pytorch so should be pretty easy to test it out with fastai!

Enjoy!

Regards,
Theodore.