Building Param Groups

champs.jaideep · April 28, 2019, 7:53am

Hi All
I am looking for a logic to build the param groups . I know we have got method split on ,that just takes network layer indices and does rest of job. I want to build using own way to enhance knowledge.
I came across method add_param_Groups in pytorch documentation

github.com

pytorch/pytorch/blob/master/torch/optim/optimizer.py#L44




    if isinstance(params, torch.Tensor):
        raise TypeError("params argument given to the optimizer should be "
                        "an iterable of Tensors or dicts, but got " +
                        torch.typename(params))


    self.state = defaultdict(dict)
    self.param_groups = []


    param_groups = list(params)
    if len(param_groups) == 0:
        raise ValueError("optimizer got an empty parameter list")
    if not isinstance(param_groups[0], dict):
        param_groups = [{'params': param_groups}]


    for param_group in param_groups:
        self.add_param_group(param_group)


def __getstate__(self):
    return {
        'defaults': self.defaults,

Here is high level logic i can think of
Assume we want to build resnet model groups (m[0] m[6],m[1])
Here can be the logic

Take children of group1 m[0] m[6]
build dict p with key [‘params’]
such that it contains list of weight params of all the children in the layer
call add_param_group with this new dict whose params contain list of all the weight params say {params [ [] [] [] ] }
similarly do for m[1]

if above logic is good please help writing the code…