Saving and loading data and models

fastai
21

Hi,

. I have created a custom databunch which I am trying to load using load_data. But I am getting an attribute error -

File “/home/views.py”, line 641, in get
path, r"/home/data_save.pkl")
File “/usr/local/lib/python3.7/site-packages/fastai/basic_data.py”, line 281, in load_data
ll = torch.load(source, map_location=‘cpu’) if defaults.device == torch.device(‘cpu’) else torch.load(source)
File “/usr/local/lib/python3.7/site-packages/torch/serialization.py”, line 529, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File “/usr/local/lib/python3.7/site-packages/torch/serialization.py”, line 702, in _legacy_load
result = unpickler.load()
AttributeError: Can’t get attribute ‘RobertaTextList’ on <module ’ main ’ from ‘manage.py’>

The RobertaTextList has been defined in the program but I am still getting the error.

Maybe I have to define this function or import it in the context that I’m loading the databunch. But I don’t know how.

Can anyone help me with this?

22

I see the same thing on the difference in size between the pth and pkl files.image

23

@sgugger

Wanted to ask small thing? I didn’t save the model and use it for prediction on test set vs I export the model,then load it and use it for prediction on test set. Both scenarios give me different accuracy with first one having accuracy of 73% and second one having accuracy of 65%

Any explanation on this or I am missing something?

24

Yeah, we can use save/load for training continuation but how can we train the model on newer data which is similar but with fewer classes, doing the same task but with just one reduced class?