Lesson 10_NLP

[robocup]

Hello,
In Lesson 10_nlp, in the Saving and Loading Models section, when I run

learn.save(‘1epoch’)

I get the following error:

---

OSError Traceback (most recent call last)
in
----> 1 learn.save(‘1epoch’)

/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in save(self, file, **kwargs)
282 @delegates(save_model)
283 def save(self, file, **kwargs):
→ 284 file = join_path_file(file, self.path/self.model_dir, ext=‘.pth’)
285 save_model(file, self.model, getattr(self,‘opt’,None), **kwargs)
286 return file

/opt/conda/envs/fastai/lib/python3.8/site-packages/fastcore/xtras.py in join_path_file(file, path, ext)
177 “Return path/file if file is a string or a Path, file otherwise”
178 if not isinstance(file, (str, Path)): return file
→ 179 path.mkdir(parents=True, exist_ok=True)
180 return path/f’{file}{ext}’
181

/opt/conda/envs/fastai/lib/python3.8/pathlib.py in mkdir(self, mode, parents, exist_ok)
1282 self._raise_closed()
1283 try:
→ 1284 self._accessor.mkdir(self, mode)
1285 except FileNotFoundError:
1286 if not parents or self.parent == self:

OSError: [Errno 30] Read-only file system: ‘/storage/data/imdb/models’

So I did the following to change the saving directory:

learn.model_dir = Path(‘/fastbook/models/’)
learn.save(‘1epoch’)

That fixed the issue, but when I navigate the /fastbook/models/ directory, I cannot find 1epoch.pth, even though somehow I am able to load it afterwards with:

learn = learn.load(‘1epoch’)

So my questions are:

1. Is there a better way to solve the initial issue above?
2. How come I cannot find 1epoch.pth but the load() function can? What is going on here?

[manju-dev]

Did you try to printing the paths?

print(Path(’/fastbook/models/’).absolute())
print(learn.model_dir)

Also try passing the path to model_dir at the time of creating learner intead of setting it after fact.

[robocup]

print(Path(’/fastbook/models/’).absolute())

gives me:

/fastbook/models

And

print(learn.model_dir)

gives me

models

Not sure which ‘models’ directory this is.
But if I do

print(Path(learn.model_dir).absolute())

I get:

/notebooks/fastbook/models

So I guess learn.model_dir is /fastbook/models, yet somehow, learn.save() tries to save in Read-only ‘/storage/data/imdb/models’ ??

[manju-dev]

@robocup

learn.model_dir = Path(’/fastbook/models/’)

This seems like it has no effect.

model.save is using the inferrred directory (at the time of creation) to save and load model.

Try setting the path you want while initializing the model by passing path to ‘model_dir’ parameter.

class Learner

[robocup]

It does have an effect. That’s why learn.save() is working. The trouble is that I can’t find the file in the directory (but learn.load() can).
I have now also tried what you suggested (passing model_dir at the creation of the learner) and I get the exact same result: learn.save() is working, but I can’t find the 1epoch.pth file in the directory afterwards (but learn.load() can).
This is so weird. Why/where is the file hiding?

[manju-dev]

are you providing the full path (from root) or just a relative path? Here is my notebook for reference:

[muellerzr (Zachary Mueller)]

@robocup

model_dir

You don’t want model_dir. You want learn.path. Model dir is only the model directory, not the save directory. Rewritten we could think of the saved path as learn.path/learn.model_dir/name. So instead try this:

learn.path = Path('fastbook')

This should then save to fastbook/models

[robocup]

Here is what I did:

And here is what I got:

Yet I can’t find the 1epoch.pth file saved in that directory…

[robocup]

As you suggested, I did:

learn.path = Path(‘fastbook’)

right before

learn.save(‘1epoch’)

and I get the exact same result: 1epoch.pth is nowhere to be found in /fastbook/models/.
Are you guys able to actually see 1epoch.pth sitting in a directory after you’ve saved? That is my issue.

[manju-dev]

You used the Path(). Did you try passing just a full string like in my screenshot?

[muellerzr (Zachary Mueller)]

Where are you trying to save it to? What’s the platform? From what it sounds like you may be on Paperspace or something where only some data points can be written to

[robocup]

I just did, and still the same issue…

[robocup]

Yes I’m using Paperspace. I am trying to save in /fastbook/models directory (cloud); which I was able to do before in Lesson 9 where I saved nn.pth by just doing:

learn.save(‘nn’)

And it saved it in /fastbook/models directory, and I can still see nn.pth in there.

Mind you, in Lesson 9 I didn’t even have to fix learn.path or model_dir beforehand. Just calling learn.save(‘nn’) saved the file in /fastbook/models directory. In Lesson 10, when I just call learn.save(‘1epoch’) without fixing learn.path or model_dir beforehand, it tries to save in /storage/data/imdb/models instead. Why there? Why not in /fastbook/models as was the case in Lesson 9?

[muellerzr (Zachary Mueller)]

@robocup

ata/imdb/models instead. Why there? Why not in /fastbook/models as was the case in Lesson 9?

Because that’s your data path. Chapter 9 had you download from kaggle, so it went to a different base directory. learn.path is based on where your data was originally downloaded to

[manju-dev]

There is some confusion about paths and being on Paperspace is adding to that confusion (as @muellerzr pointed out)

In paperspace, your pesistant storage is mounted on /storage. So you should include that in your path string.

Did you include that in your latest attempt? Post the code/screenshot (always do this when you change something)

[robocup]

In Lesson 9 the Kaggle data is downloaded in ‘/storage/archive/bluebook’ not in ‘/fastbook/models’; yet learn.save() saved in the latter.

[robocup]

I guess it’s probably an issue with Paperspace then. Because, in Lesson 9 the Kaggle data is downloaded in ‘/storage/archive/bluebook’ not in ‘/fastbook/models’; yet learn.save() saved in the latter.
And when I try to save in /storage, as you suggested, it works! (meaning I can indeed see 1epoch.pth sitting there afterwards)



But I still have the lingering (though I guess now sort of subsidiary) question: where was that 1epoch.pth being saved before? I couldn’t find it where I was saving it, but somehow learn.load() was able to load it. This is still a mystery to me.

By the way, is there a better/quicker way to attach screenshots here other than printScreen+paint+save somewhere in my machine + upload here ?