Platform: Amazon SageMaker - AWS

Part 1 (2020)
64

Thank you for this! I was struggling for a few weeks to make it work with Sagemaker v2 and finally found your template.

I was able to deploy the model to an endpoint and successfully test it. The only thing I’m still fighting is making this work with batch transform. I am trying to run a batch transform on images saved on S3, but I guess I need to modify the serve.py script to make it work, as it expects a URL as a JSON and not the image itself (which is what I think batch transform is doing).

With the original serve.py I would get this error:
2021-03-23T18:01:49.949:[sagemaker logs]: pet-classifier-test/pets-input/himalayan-cats-and-kittens-1.jpg: Requested unsupported ContentType in content_type: None

I tried modifying the input_fn function in serve.py like this:

def input_fn(request_body, content_type=JPEG_CONTENT_TYPE): logger.info('Deserializing the input data.') # process an image uploaded to the endpoint if content_type == JPEG_CONTENT_TYPE: return open_image(io.BytesIO(request_body)) # process a URL submitted to the endpoint if content_type == JSON_CONTENT_TYPE: logger.debug(f'Request body is: {request_body}') request = json.loads(request_body) logger.debug(f'Loaded JSON object: {request}') url = request['url'] logger.info(f'Downloading image from URL: {url}') img_content = requests.get(url).content logger.info(f'Returning image bytes') return io.BytesIO(img_content).read() raise Exception('Requested unsupported ContentType in content_type: {}'.format(content_type))

But I get the same error. Any hints would be appreciated. I will let you know if I’m successful at solving this.

65

im getting the following error when trying to spin up the predictor endpoint

 raise VersionConflict(dist, req).with_context(dependent_req)
cw28jf5snh-algo-1-aihs9 | pkg_resources.VersionConflict: (fastprogress 0.1.21 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('fastprogress>=0.2.1'))

Here’s the full output

[2021-04-12 15:42:18 +0000] [31] [ERROR] Error handling request /ping
ro7acc7tbr-algo-1-bv3eb | Traceback (most recent call last):
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/sagemaker_containers/_modules.py", line 246, in import_module
ro7acc7tbr-algo-1-bv3eb |     module = importlib.import_module(name)
ro7acc7tbr-algo-1-bv3eb |   File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
ro7acc7tbr-algo-1-bv3eb |     return _bootstrap._gcd_import(name[level:], package, level)
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/app.py", line 14, in <module>
ro7acc7tbr-algo-1-bv3eb |     from fastai.text import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/text/__init__.py", line 1, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .. import basics
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/basics.py", line 1, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .basic_train import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py", line 2, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .torch_core import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/torch_core.py", line 2, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .imports.torch import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/imports/__init__.py", line 1, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .core import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/imports/core.py", line 30, in <module>
ro7acc7tbr-algo-1-bv3eb |     pkg_resources.require("fastprogress>=0.2.1")
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 900, in require
ro7acc7tbr-algo-1-bv3eb |     needed = self.resolve(parse_requirements(requirements))
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 791, in resolve
ro7acc7tbr-algo-1-bv3eb |     raise VersionConflict(dist, req).with_context(dependent_req)
ro7acc7tbr-algo-1-bv3eb | pkg_resources.VersionConflict: (fastprogress 0.1.21 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('fastprogress>=0.2.1'))
ro7acc7tbr-algo-1-bv3eb | 
ro7acc7tbr-algo-1-bv3eb | During handling of the above exception, another exception occurred:
ro7acc7tbr-algo-1-bv3eb | 
ro7acc7tbr-algo-1-bv3eb | Traceback (most recent call last):
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
ro7acc7tbr-algo-1-bv3eb |     self.handle_request(listener_name, req, client, addr)
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
ro7acc7tbr-algo-1-bv3eb |     addr)
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
ro7acc7tbr-algo-1-bv3eb |     respiter = self.wsgi(environ, resp.start_response)
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/sagemaker_pytorch_container/serving.py", line 103, in main
ro7acc7tbr-algo-1-bv3eb |     user_module = modules.import_module(serving_env.module_dir, serving_env.module_name)
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/sagemaker_containers/_modules.py", line 251, in import_module
ro7acc7tbr-algo-1-bv3eb |     six.reraise(_errors.ImportModuleError, _errors.ImportModuleError(e), sys.exc_info()[2])
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/six.py", line 692, in reraise
ro7acc7tbr-algo-1-bv3eb |     raise value.with_traceback(tb)
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/sagemaker_containers/_modules.py", line 246, in import_module
ro7acc7tbr-algo-1-bv3eb |     module = importlib.import_module(name)
ro7acc7tbr-algo-1-bv3eb |   File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
ro7acc7tbr-algo-1-bv3eb |     return _bootstrap._gcd_import(name[level:], package, level)
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
ro7acc7tbr-algo-1-bv3eb |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/app.py", line 14, in <module>
ro7acc7tbr-algo-1-bv3eb |     from fastai.text import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/text/__init__.py", line 1, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .. import basics
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/basics.py", line 1, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .basic_train import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py", line 2, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .torch_core import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/torch_core.py", line 2, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .imports.torch import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/imports/__init__.py", line 1, in <module>
ro7acc7tbr-algo-1-bv3eb |     from .core import *
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/fastai/imports/core.py", line 30, in <module>
ro7acc7tbr-algo-1-bv3eb |     pkg_resources.require("fastprogress>=0.2.1")
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 900, in require
ro7acc7tbr-algo-1-bv3eb |     needed = self.resolve(parse_requirements(requirements))
ro7acc7tbr-algo-1-bv3eb |   File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 791, in resolve
ro7acc7tbr-algo-1-bv3eb |     raise VersionConflict(dist, req).with_context(dependent_req)
ro7acc7tbr-algo-1-bv3eb | sagemaker_containers._errors.ImportModuleError: (fastprogress 0.1.21 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('fastprogress>=0.2.1'))

my requirements file contains

fastai==1.0.61
fastprogress==0.2.1

So i should be using fastprogress 0.2.1

Running the Pytorch model function works fine

from sagemaker.pytorch.estimator import PyTorch 
estimator = PyTorch(entry_point='app.py',
                         source_dir='source',
                         base_job_name='***',
                         role=****',
                         framework_version='1.0.0',
                         py_version='py3',
                         instance_count=1,
                         instance_type="local")

estimator.fit(data_location)

However, this error occurs when i running the following

predictor = estimator.deploy(initial_instance_count=1, instance_type="local")

I’ve look at this stackoverflow post with no success

66

Hi guys,

I’m was not able to deploy to Sagemaker today or yesterday. Around week ago it worked.

The problem is that fastai -kernel never shows up to notebook pages (or within 30 minutes) .
I guess this command is not executed successfully in the scrips:
“conda update -n base -c defaults conda -y”

In “LifecycleConfigOnCreate” logs this shows up and nothing comes after it:

2021-05-03T09:05:06.346+03:00 Updating conda

2021-05-03T09:05:25.352+03:00 Collecting package metadata (current_repodata.json): …working… done

2021-05-03T09:06:42.368+03:00 Solving environment: …working… done

2021-05-03T09:07:05.373+03:00 ## Package Plan ## environment location: /home/ec2-user/anaconda3 added / updated specs: - conda

2021-05-03T09:07:05.373+03:00 The following packages will be downloaded: package | build ---------------------------|----------------- anyio-2.2.0 | py37h06a4308_1 123 KB ca-certificates-2021.4.13 | h06a4308_1 114 KB certifi-2020.12.5 | py37h06a4308_0 141 KB cryptography-3.4.7 | py37hd23ed53_0 904 KB docutils-0.17 | py37h06a4308_1 686 KB jupyter-packaging-0.7.12 | pyhd3eb1b0_0 14 KB jupyter_server-1.4.1 | py37h06a4308_0 311 KB jupyterlab-3.0.14 | pyhd3eb1b0_1 3.5 MB jupyterlab_server-2.4.0 | pyhd3eb1b0_0 37 KB lxml-4.6.3 | py37h9120a33_0 1.2 MB nbclassic-0.2.6 | pyhd3eb1b0_0 19 KB openssl-1.1.1k | h27cfd23_0 2.5 MB pip-21.0.1 | py37h06a4308_0 1.8 MB scipy-1.6.2 | py37h91f5cce_0 15.4 MB sniffio-1.2.0 | py37h06a4308_1 15 KB urllib3-1.26.4 | pyhd3eb1b0_0 105 KB ------------------------------------------------------------ Total: 26.8 MB

2021-05-03T09:07:05.374+03:00 The following NEW packages will be INSTALLED: anyio pkgs/main/linux-64::anyio-2.2.0-py37h06a4308_1 colorama pkgs/main/noarch::colorama-0.4.4-pyhd3eb1b0_0 cryptography pkgs/main/linux-64::cryptography-3.4.7-py37hd23ed53_0 docutils pkgs/main/linux-64::docutils-0.17-py37h06a4308_1 jupyter-packaging pkgs/main/noarch::jupyter-packaging-0.7.12-pyhd3eb1b0_0 jupyter_server pkgs/main/linux-64::jupyter_server-1.4.1-py37h06a4308_0 jupyterlab pkgs/main/noarch::jupyterlab-3.0.14-pyhd3eb1b0_1 jupyterlab_server pkgs/main/noarch::jupyterlab_server-2.4.0-pyhd3eb1b0_0 lxml pkgs/main/linux-64::lxml-4.6.3-py37h9120a33_0 nbclassic pkgs/main/noarch::nbclassic-0.2.6-pyhd3eb1b0_0 pip pkgs/main/linux-64::pip-21.0.1-py37h06a4308_0 requests pkgs/main/noarch::requests-2.25.1-pyhd3eb1b0_0 sniffio pkgs/main/linux-64::sniffio-1.2.0-py37h06a4308_1 urllib3 pkgs/main/noarch::urllib3-1.26.4-pyhd3eb1b0_0

2021-05-03T09:07:05.374+03:00 The following packages will be UPDATED: ca-certificates conda-forge::ca-certificates-2020.12.~ --> pkgs/main::ca-certificates-2021.4.13-h06a4308_1 openssl conda-forge::openssl-1.1.1j-h7f98852_0 --> pkgs/main::openssl-1.1.1k-h27cfd23_0 scipy 1.6.1-py37h91f5cce_0 --> 1.6.2-py37h91f5cce_0

2021-05-03T09:07:05.374+03:00 The following packages will be SUPERSEDED by a higher-priority channel: certifi conda-forge::certifi-2020.12.5-py37h8~ --> pkgs/main::certifi-2020.12.5-py37h06a4308_0

2021-05-03T09:07:05.374+03:00 Downloading and Extracting Packages

2021-05-03T09:07:06.374+03:00 openssl-1.1.1k | 2.5 MB | | 0% openssl-1.1.1k | 2.5 MB | ########## | 100% openssl-1.1.1k | 2.5 MB | ########## | 100% lxml-4.6.3 | 1.2 MB | | 0% lxml-4.6.3 | 1.2 MB | ########## | 100% lxml-4.6.3 | 1.2 MB | ########## | 100% sniffio-1.2.0 | 15 KB | | 0% sniffio-1.2.0 | 15 KB | ########## | 100% sniffio-1.2.0 | 15 KB | ########## | 100% cryptography-3.4.7 | 904 KB | | 0% cryptography-3.4.7 | 904 KB | ########## | 100% cryptography-3.4.7 | 904 KB | ########## | 100% certifi-2020.12.5 | 141 KB | | 0% certifi-2020.12.5 | 141 KB | ########## | 100% certifi-2020.12.5 | 141 KB | ########## | 100%

2021-05-03T09:07:07.375+03:00 jupyterlab-3.0.14 | 3.5 MB | | 0% jupyterlab-3.0.14 | 3.5 MB | ########## | 100% jupyterlab-3.0.14 | 3.5 MB | ########## | 100% urllib3-1.26.4 | 105 KB | | 0% urllib3-1.26.4 | 105 KB | ########## | 100% urllib3-1.26.4 | 105 KB | ########## | 100% jupyterlab_server-2. | 37 KB | | 0% jupyterlab_server-2. | 37 KB | ########## | 100% jupyterlab_server-2. | 37 KB | ########## | 100% pip-21.0.1 | 1.8 MB | | 0% pip-21.0.1 | 1.8 MB | ########## | 100% pip-21.0.1 | 1.8 MB | ########## | 100% nbclassic-0.2.6 | 19 KB | | 0% nbclassic-0.2.6 | 19 KB | ########## | 100% nbclassic-0.2.6 | 19 KB | ########## | 100% jupyter_server-1.4.1 | 311 KB | | 0% jupyter_server-1.4.1 | 311 KB | ########## | 100% jupyter_server-1.4.1 | 311 KB | ########## | 100%

2021-05-03T09:07:08.375+03:00 scipy-1.6.2 | 15.4 MB | | 0% scipy-1.6.2 | 15.4 MB | ####### | 71% scipy-1.6.2 | 15.4 MB | ########## | 100% scipy-1.6.2 | 15.4 MB | ########## | 100% anyio-2.2.0 | 123 KB | | 0% anyio-2.2.0 | 123 KB | ########## | 100% anyio-2.2.0 | 123 KB | ########## | 100% jupyter-packaging-0. | 14 KB | | 0% jupyter-packaging-0. | 14 KB | ########## | 100% jupyter-packaging-0. | 14 KB | ########## | 100% ca-certificates-2021 | 114 KB | | 0% ca-certificates-2021 | 114 KB | ########## | 100% ca-certificates-2021 | 114 KB | ########## | 100%

2021-05-03T09:07:09.376+03:00 docutils-0.17 | 686 KB | | 0% docutils-0.17 | 686 KB | ########## | 100% docutils-0.17 | 686 KB | ########## | 100%

2021-05-03T09:07:09.376+03:00 Preparing transaction: …working… done

2021-05-03T09:07:09.376+03:00 Verifying transaction: …working… failed

2021-05-03T09:07:09.376+03:00 The environment is inconsistent, please check the package plan carefully

2021-05-03T09:07:09.376+03:00 The following packages are causing the inconsistency: - defaults/noarch::sphinx==3.5.1=pyhd3eb1b0_0 - defaults/linux-64::scikit-image==0.17.2=py37hdf5156a_0 - defaults/noarch::numpydoc==1.1.0=pyhd3eb1b0_1 - defaults/noarch::seaborn==0.11.1=pyhd3eb1b0_0 - defaults/linux-64::secretstorage==3.3.1=py37h06a4308_0 - defaults/linux-64::conda==4.8.4=py37_0 - defaults/linux-64::scikit-learn==0.23.2=py37h0573a6f_0 - defaults/linux-64::spyder==4.2.1=py37h06a4308_1 - defaults/linux-64::_anaconda_depends==2020.07=py37_0 - defaults/linux-64::anaconda-client==1.7.2=py37_0 - defaults/noarch::anaconda-project==0.9.1=pyhd3eb1b0_1 - defaults/linux-64::scipy==1.6.1=py37h91f5cce_0 - defaults/linux-64::statsmodels==0.12.2=py37h27cfd23_0 - defaults/linux-64::keyring==22.0.1=py37h06a4308_0 - defaults/linux-64::patsy==0.5.1=py37_0 - defaults/linux-64::pyopenssl==19.1.0=py37_0

2021-05-03T09:07:09.376+03:00 ==> WARNING: A newer version of conda exists. <== current version: 4.8.4 latest version: 4.10.1

2021-05-03T09:07:09.376+03:00 Please update conda by running $ conda update -n base -c defaults conda

2021-05-03T09:07:09.376+03:00 RemoveError: ‘requests’ is a dependency of conda and cannot be removed from

2021-05-03T09:07:09.376+03:00 conda’s operating environment.

I’m using this deploy command (from fastai’s website):
https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/create/review?filter=active&templateURL=https://fastai-cfn.s3.amazonaws.com/sagemaker-cfn-course-v4.yml&stackName=FastaiSageMakerStack

Does anybody know how to fix this, or am I missing something?
I guess packages just got into a state where they won’t work together.

67

I tried tuning scripts in the Jupyter console, and as result I managed to get the fastai kernel activated.
I’m not sure exactly what were the needed actions, but adding these commands (before the
“conda update -n base -c defaults conda -y” -command )
might have had an effect:
conda install conda=4.10.1 -y
conda update --all -y

68

Hi,

I hope this is relevant to post here. I’ve created a guide for getting the fastai-v4 course setup on an AWS SageMaker instance in the EU London region. I came up against quite a few issues in terms of the ml.p2 instance types not being available in my region and the fastai kernel being missing. So I thought I’d collate all my findings and solutions in one place. I’m also very new to all of this therefore my broken down guide might be helpful to real newbies like myself :slight_smile:

Thanks,
Chelsea

69

Hello, Arizona_Sky!

Thanks for the note! I faced a similar issue later on and wound up modifying the stack template as documented here: FastaiSageMakerStack template (https://fastai-cfn.s3.amazonaws.com/sagemaker-cfn-course-v4.yml) fails updating conda due to conda env update dependencies · Issue #60 · fastai/course20 · GitHub

Agree with you that the cloud formation template should be modified on the course site to prevent repeated fails on new entrants to fastai on AWS SageMaker.

Best
PO

70

Me too

71

Hello! I am new to this course and AWS Sagemaker. I’ve been trying to launch the AWS CloudFormation specifically for Tokyo Region. I already increased my limit to 1 as instructed here: Amazon SageMaker | Practical Deep Learning for Coders

However, I am getting the error:

Screenshot 2022-03-22 024943
Screenshot 2022-03-22 024943894×128 9.35 KB

May I ask if there’s a link on how to solve this? Thanks!

72

Same here, I’m getting the same error when trying to launch from US West 1. I’ve been using my own Sagemaker instance but have run into constant issues with conflicting packages and having to re-install my packages every time I want to study. That’s costing me 20-30 minutes each session so I figured I’d try the cloudformation path. No luck yet but I’ll post here if I make any progress.

The underlying S3 file it’s trying to access is at the following location and I wonder if there is a new version or if its no longer public: https://s3-eu-west-1.amazonaws.com/mmcclean-public-files/sagemaker-fastai-notebook/sagemaker-cfn-course-v4.yaml

73

Instead of using the S3 route, I’ve decided to downloaded the yaml file from here: FastaiSageMakerStack template (https://fastai-cfn.s3.amazonaws.com/sagemaker-cfn-course-v4.yml) fails updating conda due to conda env update dependencies · Issue #60 · fastai/course20 · GitHub
and will upload the template to cloudformation. Will need to make the corrections to line 98 and also changed the instance type. This post was also helpful: AWS NVIDIA GPU Instance Setup (for UK) | mathschelsea

74

Thanks for the tip! I’ve tried what you suggested and finally got to create the stack. However, I cannot seem to find the kernel for fastai. Curious if it is showing in your notebook instance? Thank you!

75

I wasn’t either and had to make changes and recreate the stack a few times. One time, the kernel actually showed up but when I tried to connect, it would error and timeout. I never got it to work unfortunately. I’ve spent too many days trying to get it fast ai to work on Sagemaker and have been blocked on actually learning the course material so I have decided to move on to Colab. I use AWS quite a bit professionally but at this time, my focus is on learning DL so I’ll get back to Sagemaker on my next course (non-FastAI framework).

76

Got it! Thank you so much! I’ll probably just move on to Colab/Gradient (whichever works) too.

1 Like