Fastai v2 default layer initialization

etremblay · March 7, 2020, 7:52pm

I was trying to find if fastai v2 does any layer initialization for us by default. For example with nn.init.kaiming_normal. From my quick code search it doesn’t appear so, but I wanted to double check.

Thanks,

sgugger · March 7, 2020, 8:11pm

It does initialize convolution layers in ConvLayer and fastai’s Embedding also has a custom init.

etremblay · March 7, 2020, 8:15pm

Should layers in a tabular model or any Linear layer be initialized manually then as a best practice? For example with kaiming normal?

Thanks!

sgugger · March 7, 2020, 8:20pm

If you want to.

jeremy · March 8, 2020, 1:54am

I thought PyTorch might be doing reasonable init nowadays by default - worth double-checking that.

etremblay · March 8, 2020, 1:29pm

Looking at the code in pytorch, seems like it is using kaiming uniform:

github.com

pytorch/pytorch/blob/master/torch/nn/modules/linear.py

import math

import torch
from torch.nn.parameter import Parameter
from .. import functional as F
from .. import init
from .module import Module


class Identity(Module):
    r"""A placeholder identity operator that is argument-insensitive.

    Args:
        args: any argument (unused)
        kwargs: any keyword argument (unused)

    Examples::

        >>> m = nn.Identity(54, unused_argument1=0.1, unused_argument2=False)
        >>> input = torch.randn(128, 20)

This file has been truncated. show original