Request for adding pytorch optimizers support

fastai fastai dev
1

Hi,

I’m questioning whether or not it’s possible to add support for torch.optim optimizers, like:
learn = Learner(opt_func=torch.optim.Adam)

The above code return an issue:

/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py in add_param_group(self, param_group)
    198             if not isinstance(param, torch.Tensor):
    199                 raise TypeError("optimizer can only optimize Tensors, "
--> 200                                 "but one of the params is " + torch.typename(param))
    201             if not param.is_leaf:
    202                 raise ValueError("can't optimize a non-leaf Tensor")

TypeError: optimizer can only optimize Tensors, but one of the params is list
1 Like
2

The library in general will work better if you use its optimizers (and all PyTorch optimizers are inside fastai). If you absolutely need to use a PyTorch optimizer, you need to wrap it inside an OptimWrapper. Checkout the end of notebook 12_optimizer, there are examples to check the fastai’s optimizers give the same results as a PyTorch optimizer.

2 Likes
3

Brilliant. Thanks for the answer. Have a good day sir.

4

Hi again,

I have checked out the notebook you mentioned and try this way:
def opt_func(ps, lr=defaults.lr): return OptimWrapper(torch.optim.AdamW(ps, lr=lr))
learn = unet_learner(resnet34, opt_func=opt_func)

However it still faces this error:

/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py in add_param_group(self, param_group)
    198             if not isinstance(param, torch.Tensor):
    199                 raise TypeError("optimizer can only optimize Tensors, "
--> 200                                 "but one of the params is " + torch.typename(param))
    201             if not param.is_leaf:
    202                 raise ValueError("can't optimize a non-leaf Tensor")

TypeError: optimizer can only optimize Tensors, but one of the params is list

The list has length of 3 if the arch is resnet34, and 2 for xse_resnext34 (just an example).

5

You are trying to create a learner that does transfer learning with an opt_func that does not handle that. You need to pass something like [{'params': pg, 'lr': lr} for pg in ps] to your PyTorch optimizer, IIRC. Your way works only if there is one list of parameters (e.g. one parameter group that is the whole model) only.

1 Like
6

I am trying to use Ranger with resnet50 and I am getting the same error. I am referring to this notebook by @muellerzr.
Posting the code and error here. It is a multilabel classification problem.

Code:

dls = ImageDataLoaders.from_df(labels, trn_path, label_delim=';', batch_tfms=tfms, seed = 42) 
!git clone https://github.com/sdoria/mish '/kaggle/working/mish'
%cd /kaggle/working/mish
from rangerlars import *
from mish import *
from mxresnet import *
from ranger import *
opt_func = partial(Ranger, betas=(0.9,0.99), eps=1e-6)
learn = cnn_learner(dls, resnet50, metrics=partial(accuracy_multi, thresh=0.5), opt_func=opt_func, model_dir = '/kaggle')

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-19-f9b94aca0df4> in <module>
----> 1 learn = cnn_learner(dls, resnet50, metrics=partial(accuracy_multi, thresh=0.5), opt_func=opt_func, model_dir = '/kaggle')

/opt/conda/lib/python3.6/site-packages/fastcore/utils.py in _f(*args, **kwargs)
    425         log_dict = {**func_args.arguments, **{f'{k} (not in signature)':v for k,v in xtra_kwargs.items()}}
    426         log = {f'{f.__qualname__}.{k}':v for k,v in log_dict.items() if k not in but}
--> 427         inst = f(*args, **kwargs) if to_return else args[0]
    428         init_args = getattr(inst, 'init_args', {})
    429         init_args.update(log)

/opt/conda/lib/python3.6/site-packages/fastai2/vision/learner.py in cnn_learner(dls, arch, loss_func, pretrained, cut, splitter, y_range, config, n_out, normalize, **kwargs)
    175     model = create_cnn_model(arch, n_out, ifnone(cut, meta['cut']), pretrained, y_range=y_range, **config)
    176     learn = Learner(dls, model, loss_func=loss_func, splitter=ifnone(splitter, meta['split']), **kwargs)
--> 177     if pretrained: learn.freeze()
    178     return learn
    179 

/opt/conda/lib/python3.6/site-packages/fastai2/learner.py in freeze(self)
    486 
    487 @patch
--> 488 def freeze(self:Learner): self.freeze_to(-1)
    489 
    490 @patch

/opt/conda/lib/python3.6/site-packages/fastai2/learner.py in freeze_to(self, n)
    481 @patch
    482 def freeze_to(self:Learner, n):
--> 483     if self.opt is None: self.create_opt()
    484     self.opt.freeze_to(n)
    485     self.opt.clear_state()

/opt/conda/lib/python3.6/site-packages/fastai2/learner.py in create_opt(self)
    139     def _bn_bias_state(self, with_bias): return bn_bias_params(self.model, with_bias).map(self.opt.state)
    140     def create_opt(self):
--> 141         self.opt = self.opt_func(self.splitter(self.model), lr=self.lr)
    142         if not self.wd_bn_bias:
    143             for p in self._bn_bias_state(True ): p['do_wd'] = False

/kaggle/working/mish/ranger.py in __init__(self, params, lr, alpha, k, betas, eps, weight_decay)
     23         #prep defaults and init torch.optim base
     24         defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay)
---> 25         super().__init__(params,defaults)
     26 
     27         #now we can get to work...

/opt/conda/lib/python3.6/site-packages/torch/optim/optimizer.py in __init__(self, params, defaults)
     49 
     50         for param_group in param_groups:
---> 51             self.add_param_group(param_group)
     52 
     53     def __getstate__(self):

/opt/conda/lib/python3.6/site-packages/torch/optim/optimizer.py in add_param_group(self, param_group)
    198             if not isinstance(param, torch.Tensor):
    199                 raise TypeError("optimizer can only optimize Tensors, "
--> 200                                 "but one of the params is " + torch.typename(param))
    201             if not param.is_leaf:
    202                 raise ValueError("can't optimize a non-leaf Tensor")

TypeError: optimizer can only optimize Tensors, but one of the params is list
7

You should use ranger in the fastai2 library. It’s simply ‘ranger’. Note that that notebook only works in fastaiv1, not fastaiv2. You may want to look at my SOTA notebook in the Practical Deep Learning 2.0 repo for Mish, Ranger, etc :slight_smile: @dipam7

2 Likes
8

Thanks, I’ll take a look

9

Note that using a PyTorch optimizer is documented in this tutorial.

2 Likes