How to save FAST AI resnet50 model to Google Drive after training in google colab notebook

Shiva_K · July 14, 2019, 6:01pm

Hi,

So I trained a resnet34 model on google colab using fastai. I then used learn.save(‘resnet-model’) to save . Turns out this model is in “path_where_data_sits”/models. Also learn.export() exports this to “path_where_data_sits”/export.pkl. How do I save this model to google drive? I tried

from google.colab import files
files.download(’/root/.fastai/data/planet/export.pkl’)

I am not sure if this works. The reason I want this model saved in a file is that I want to use it to deploy a web application on a server. In Pytorch I would do something like

model_save_name = ‘classifier.pt’
path = F"/content/gdrive/My Drive/{model_save_name}"
torch.save(model.state_dict(), path)

How do I persist my trained model, I am thinking the learn.export() creates a file with all the classes, the weights and everything. I just need to somehow persist this for later use. Please help.

Here is a link to my notebook Amazon planet challenge

The notebook has all the places where the files are downloaded and all, and their respective paths, please help me in saving this model to drive.

Thank you so much.

muellerzr · July 14, 2019, 6:16pm

You should be able to adjust learn.path() to the path where you want to save/export it to and that should solve the problem.

Eg
learn.path = Path(‘’) will save it to your base directory.

Edit: It is learn.path that you need to change. Not learn.model.path. My bad!

Shiva_K · July 15, 2019, 9:09am

So I saved my resent-50 model after achieving an accuracy of 93% as follows

learn.export(‘train_resnet_model_50.pkl’). My learner is a cnn_learner(data, arch, metric=[acc_02, fbeta]) where arch = models.resent50.

Upon inspecting my path.ls() I get the fillowing

[PosixPath(’/root/.fastai/data/planet/train_v2.csv.zip’),
PosixPath(’/root/.fastai/data/planet/train_resnet_model_50.pkl’),
PosixPath(’/root/.fastai/data/planet/train-jpg.tar.7z’),
PosixPath(’/root/.fastai/data/planet/train-jpg’),
PosixPath(’/root/.fastai/data/planet/__MACOSX’),
PosixPath(’/root/.fastai/data/planet/models’),
PosixPath(’/root/.fastai/data/planet/train_v2.csv’)]

the second path seems to be the one where the model lives. How do I upload this file to google drive? I know this is absolutely trivial, But I also know that I don’t know how to. Like this is what I tried, I am not sure if this is the correct approach though?

from google.colab import files
files.download(’/root/.fastai/data/planet/train_resnet_model_50.pkl’), this downloads the file to my local machine but with an error that reads:

Exception happened during processing of request from (’::ffff:127.0.0.1’, 34680, 0, 0)
Traceback (most recent call last):
File “/usr/lib/python3.6/socketserver.py”, line 320, in _handle_request_noblock
self.process_request(request, client_address)
File “/usr/lib/python3.6/socketserver.py”, line 351, in process_request
self.finish_request(request, client_address)
File “/usr/lib/python3.6/socketserver.py”, line 364, in finish_request
self.RequestHandlerClass(request, client_address, self)
File “/usr/lib/python3.6/socketserver.py”, line 724, in init
self.handle()
File “/usr/lib/python3.6/http/server.py”, line 418, in handle
self.handle_one_request()
File “/usr/lib/python3.6/http/server.py”, line 406, in handle_one_request
method()
File “/usr/lib/python3.6/http/server.py”, line 639, in do_GET
self.copyfile(f, self.wfile)
File “/usr/lib/python3.6/http/server.py”, line 800, in copyfile
shutil.copyfileobj(source, outputfile)
File “/usr/lib/python3.6/shutil.py”, line 82, in copyfileobj
fdst.write(buf)
File “/usr/lib/python3.6/socketserver.py”, line 803, in write
self._sock.sendall(b)
ConnectionResetError: [Errno 104] Connection reset by peer

But after running that piece of code, I do have a file on my local machine named “train_resnet_model_50.pkl”. So I went ahead and uploaded this to my Google Drive by right-clicking on the My Drive folder and choosing the aforementioned file on my machine. Is this the right way to get a fastai model from colab into my google drive? The exception makes me really skeptical. Thanks a lot for helping me out, appreciate it.

muellerzr · July 15, 2019, 10:12am

@Shiva_K, not quite what I mean, I’ll show some photos

We’ll use the lesson 4 - tabular as an example but it’s true for all of them.

The above describes the first step, changing the model’s path, and you can see on the left I have the ‘Files’ tab open. (If you don’t have this, look at the far left corner and you should see a little white arrow on a black tab pointing right)

Now I can run learn.export() and it’s in those files! Last thing is to right-mouse-click on the file, and select “Download”. Couldn’t quite get an image of this, but should be self explanatory. Then you can upload it to your google drive.

Hope that helps

Shiva_K · July 15, 2019, 11:41am

Ah yes! Got it. Thank you so much. Your the best.

muellerzr · July 15, 2019, 11:53am

No problem! Glad I could help

NicWick · July 16, 2019, 10:20am

The recommended way by the course v3 instructions from https://course.fast.ai/start_colab.html is

from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)
root_dir = "/content/gdrive/My Drive/"
base_dir = root_dir + 'fastai-v3/'

And then append base_dir to the file paths included in the notebook.

This is what I am doing and it works well.

Shiva_K · July 31, 2019, 1:25pm

Oh yes, I saw that, makes sense.

MaxA · October 2, 2019, 4:14am

Step 1 Mount your drive

from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)
root_dir = "/content/gdrive/My Drive/"
base_dir = root_dir + 'fastai-v3/'

Step 2 - Create a variable with your folder name and create a folder on Google Drive
Here’s an example for Lesson 1

dest=Path(base_dir + "models/oxford-iiit-pet/")
try:
  dest.mkdir(parents=True, exist_ok=False)
except FileExistsError:
  print ('File Already Exists')

Step 3 - Update the destination anywhere your code saves your models
For example, for lesson 1, change

learn.save('stage-1')

to

learn.save(dest/'stage-1')

Other recommendations describe how

to save both data and models to Google Drive
to copy models manually by downloading them to your local drive first.

Unless you are creating your custom dataset, I recommend against the first option. Because to retrain the model, you’ll need to reference data anyway, I recommend (re)downloading data directly to the Collab machine

I did not like the second option because it’s manual.

Here’s a link to my Collab notebook for Lesson 1

wcneill · March 22, 2020, 10:39pm

I don’t know how many people use Google Colab as apposed to the other GPU options, but I would say that this is a really important thread. I feel like it either needs to be a sticky, or these directions posted in the “set-up” section of the course page.

Thank you @MaxA and @muellerzr for two great ways to get this done. You really helped me get set-up for this course on Colab.

sahil1234 · October 29, 2021, 8:15am

Hi!
The saving part is all nice
However the loading part by learn.save(dest/'filename') leads to an error in Google Colab (and not in Paperspace Gradient)

RuntimeError: Error(s) in loading state_dict for SequentialRNN:
	size mismatch for 0.encoder.weight: copying a param with shape torch.Size([40504, 400]) from checkpoint, the shape in current model is torch.Size([40288, 400]).
	size mismatch for 0.encoder_dp.emb.weight: copying a param with shape torch.Size([40504, 400]) from checkpoint, the shape in current model is torch.Size([40288, 400]).
	size mismatch for 1.decoder.weight: copying a param with shape torch.Size([40504, 400]) from checkpoint, the shape in current model is torch.Size([40288, 400]).
	size mismatch for 1.decoder.bias: copying a param with shape torch.Size([40504]) from checkpoint, the shape in current model is torch.Size([40288]).


What do you think is causing this problem