Can't pickle local object

I am getting an unusual error when I try to export a learner.

Code to make learner and data:

data = (PointsItemList.from_df(df, path='./', cols='image')
        .transform(tfms, tfm_y=True, size=300, padding_mode='reflection', remove_out=False)

learner = cnn_learner(

After the model has been trained, I run:'stage1',return_path=True)

Which works fine. Then I run:


This is the error I get:

AttributeError Traceback (most recent call last)

[<ipython-input-15-de10b3ec63b2>]( in <module>() ----> 1 learner.export('model.pkl')

3 frames

[/usr/local/lib/python3.6/dist-packages/fastai/]( in export(self, file, destroy) 241 state['data'] =**xtra) 242 state['cls'] = self.__class__ --> 243 try_save(state, self.path, file) 244 if destroy: self.destroy() 245

[/usr/local/lib/python3.6/dist-packages/fastai/]( in try_save(state, path, file) 414 #To avoid the warning that come from PyTorch about model not being checked 415 warnings.simplefilter("ignore") --> 416, target) 417 except OSError as e: 418 raise Exception(f"{e}\n Can't write {path/file}. Pass an absolute writable pathlib obj `fname`.")

[/usr/local/lib/python3.6/dist-packages/torch/]( in save(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization) 326 327 with _open_file_like(f, 'wb') as opened_file: --> 328 _legacy_save(obj, opened_file, pickle_module, pickle_protocol) 329 330

[/usr/local/lib/python3.6/dist-packages/torch/]( in _legacy_save(obj, f, pickle_module, pickle_protocol) 399 pickler = pickle_module.Pickler(f, protocol=pickle_protocol) 400 pickler.persistent_id = persistent_id --> 401 pickler.dump(obj) 402 403 serialized_storage_keys = sorted(serialized_storages.keys())

AttributeError: Can't pickle local object 'TorchHistory.add_log_hooks_to_pytorch_module.<locals>.<lambda>' 

We’ll need more information than that to help. How are you building the Learner? How are you building the Data?

Also, for code please wrap it in three ` 's so it’s readable, like I have done so below:


Thanks! I’ve added more detail.

I believe it’s complaining about you’re using a lambda function, but I don’t see one in the code you provided. If you’re using a getx or gety lambda in your data loader, then replace it with a function and you’ll be ok.

That’s curious. The only thing I have in the labeling function is

def is_validation(imgpath):
    if imgpath.endswith('_val.jpeg'):
      return True
      return False

Are you using a WandB callback?

TorchHistory is referring to that callback apparently. Check source code here.

If you are, just remove the callback before pickling.


Ah, that must be it! Thanks so much! I didn’t consider it because it’s not part of the model creation code.

Yes, fastai will pickle the Learner objects which includes callbacks, loss function, metrics, etc.

@nickvu how did you detach the WandbCallback? I’m facing a very similar error but with TorchGraph rather than TorchHistory. Just like you, I’m not initialising the Learner with WandbCallback but using it only when calling

When I look at I don’t see WandbCallback there as it detaches itself after fitting.

cc @ilovescience

Thanks for your help!

1 Like

Full error message:

AttributeError                            Traceback (most recent call last)
<ipython-input-57-c9af5cd1fcdd> in <module>
----> 1 learn.export(fname=f"{fname}.pkl")

~/miniconda3/envs/stable/lib/python3.8/site-packages/fastai/ in export(self, fname, pickle_protocol)
    538         #To avoid the warning that come from PyTorch about model not being checked
    539         warnings.simplefilter("ignore")
--> 540, self.path/fname, pickle_protocol=pickle_protocol)
    541     self.create_opt()
    542     if state is not None: self.opt.load_state_dict(state)

~/.local/lib/python3.8/site-packages/torch/ in save(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization)
    362         if _use_new_zipfile_serialization:
    363             with _open_zipfile_writer(opened_file) as opened_zipfile:
--> 364                 _save(obj, opened_zipfile, pickle_module, pickle_protocol)
    365                 return
    366         _legacy_save(obj, opened_file, pickle_module, pickle_protocol)

~/.local/lib/python3.8/site-packages/torch/ in _save(obj, zip_file, pickle_module, pickle_protocol)
    464     pickler = pickle_module.Pickler(data_buf, protocol=pickle_protocol)
    465     pickler.persistent_id = persistent_id
--> 466     pickler.dump(obj)
    467     data_value = data_buf.getvalue()
    468     zip_file.write_record('data.pkl', data_value, len(data_value))

AttributeError: Can't pickle local object 'TorchGraph.hook_torch_modules.<locals>.backward_hook'
1 Like

Did anyone find a solution for this? As @rsomani95 said, I too could not remove the callback as does not show the WandbCallback

use dill e.g.

def save_for_meta_learning(args, ckpt_filename=''):
    if is_lead_worker(args.rank):
        import dill
        # - ckpt
        assert uutils.xor(args.training_mode == 'epochs', args.training_mode == 'iterations')
        args_pickable = uutils.make_args_pickable(args)
        # args.meta_learner.args = args_pickable
        f: nn.Module = get_model_from_ddp(args.base_model)
        # pickle vs{'training_mode': args.training_mode,  # its or epochs
                    'epoch_num': args.epoch_num,
                    # 'args': args_pickable,
                    'args_pickable': args_pickable,
                    # 'meta_learner': args.meta_learner,
                    'meta_learner_str': str(args.meta_learner),
                    # 'f': f,
                    'f_state_dict': f.state_dict(),
                    'f_str': str(f),
                    # 'f_modules': f._modules,
                    # 'f_modules_str': str(f._modules),
                    'outer_opt_state_dict': args.outer_opt.state_dict()
                   f=args.log_root / ckpt_filename)