Occassionally SageMaker gets stuck in a redirection loop when trying to connect to the Notebook Instance.
I’m not sure what causes the issue, but it seems to happen after shutting an instance down, then restarting at a later point. Perhaps related to unsaved notebooks?
Anyway the error log is here:
[E 21:52:27.200 NotebookApp] Uncaught exception POST /sagemaker/internal/agent/git/clone/poll?notebookInstanceArn=arn:aws:sagemaker:ap-southeast-2:120793027141:notebook-instance/fastai-v4 (10.0.1.24)
HTTPServerRequest(protocol=‘https’, host=‘10.0.125.69:8443’, method=‘POST’, uri=‘/sagemaker/internal/agent/git/clone/poll?notebookInstanceArn=arn:aws:sagemaker:ap-southeast-2:120793027141:notebook-instance/fastai-v4’, version=‘HTTP/1.1’, remote_ip=‘10.0.1.24’)
Traceback (most recent call last):
File “/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/tornado/web.py”, line 1702, in _execute
result = method(*self.path_args, **self.path_kwargs)
File “/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/sagemaker_nbi_agent/base.py”, line 38, in post
response = self.do_post(request)
File “/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/sagemaker_nbi_agent/git/git.py”, line 172, in do_post
overall_status = self.get_process_status(pid)
File “/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/sagemaker_nbi_agent/git/git.py”, line 199, in get_process_status
overall_status = self._get_clone_end_status()
File “/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/sagemaker_nbi_agent/git/git.py”, line 209, in _get_clone_end_status
with open(os.path.join(‘/tmp/.sagemaker’, ‘clone_statuses’)) as f:
FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/.sagemaker/clone_statuses’
And I just fixed it…!
But connecting to JupyterLab first, and then going back to Jupyter it seems to be working again.