'invalid magic number' when loading saved pickled module from Scaling Up: Road to the Top, Part 3

Part 2 2022/23
1

I’ve downloaded the pickle file that was generated while executing in the following notebook:

I want to load this file and run the predictions locally as an exercise. However, when loading the pickled model I’m getting an error:

File “Lesson7/l7_predict.py”, line 7, in
learn = load_learner(os.path.join(os.getcwd(), pickleFile))
File “/home/tromanow/HF/.env/lib/python3.8/site-packages/fastai/learner.py”, line 446, in load_learner
try: res = torch.load(fname, map_location=map_loc, pickle_module=pickle_module)
File “/home/tromanow/HF/.env/lib/python3.8/site-packages/torch/serialization.py”, line 815, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File “/home/tromanow/HF/.env/lib/python3.8/site-packages/torch/serialization.py”, line 1035, in _legacy_load
raise RuntimeError(“Invalid magic number; corrupt file?”)
RuntimeError: Invalid magic number; corrupt file?

2

I’ve re-uploaded the pickled file to kaggle input and tried re-loading it in the same kaggle notebook where it was saved. I’m getting the same error:

tta_res = load_learner(‘/kaggle/input/saved-trained-model/tta_res.pkl’)

How do I load the learner that is saved in the following line:

save_pickle(‘tta_res.pkl’, tta_res)

In this notebook:

3

So in the notebook you linked, tta_res is an array defined as tta_res = [] and then populated with the TTA predictions, so it’s a list of TTA predictions, not a model.

You can save any python object with save_pickle but you can only load learner objects with load_learner. In this case, you try to load a list with load_learner, which is the reason why it fails. If you want to load the array from the tta_res.pkl file, you can use load_pickle.

1 Like
4

I’m having a hard time importing this function. So far I’ve tried the following, none work:
from fastai import load_pickle
from fastcore import load_pickle
from fastkaggle import load_pickle

5

You can import fastai’s load_pickle function as follows:

from fastcore.all import *

Doing so in colab:

image
image898×300 15.8 KB

You can also just import load_pickle:

image
image880×296 16 KB