Setting up fastai on the Sagemaker is a very smooth process and works well! Thank you!
I wonder why the following code results in re-download of resnet34 every time I restart the notebook? In’s storage persistent?
data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
the sagemaker tmp directory is reset on restart, make sure your data path doesnt cache things there
Everything not under
~/SageMaker is not saved between notebook restarts. You need to remap the
~/.fastai directory to somewhere on
~/SageMaker. Some example scripts to run on notebook creation are:
mkdir /home/ec2-user/SageMaker/.torch && ln -s /home/ec2-user/SageMaker/.torch /home/ec2-user/.torch
mkdir /home/ec2-user/SageMaker/.fastai && ln -s /home/ec2-user/SageMaker/.fastai /home/ec2-user/.fastai
Every time you start the notebook you will need to recreate the symlinks with the commands:
ln -s /home/ec2-user/SageMaker/.torch /home/ec2-user/.torch
ln -s /home/ec2-user/SageMaker/.fastai /home/ec2-user/.fastai
The SageMaker setup notes described here do this for you automatically.