Sagemaker Notebook deployment problem - no FastAi kernel

Part 1 (2020)
1

I’m deploying to Sagemaker through these instructions to a p3 instance. The notebook is created fine but when I’m prompted for a kernel, there is no fastai option.

I’ve tried using the fix at the end of this thread.

Here are the logs from my LifeCycleConfigOnCreate. Will keep trying to debug, but any help is appreciated!

2021-06-02T21:07:33.523-04:00	Starting on Create script

2021-06-02T21:07:34.523-04:00	Finishing on Create script

2021-06-02T21:07:34.523-04:00	Creating dirs and symlinks

2021-06-02T21:07:34.523-04:00	Updating conda

2021-06-02T21:07:59.529-04:00	Collecting package metadata (current_repodata.json): ...working... done

2021-06-02T21:09:34.549-04:00	Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.

2021-06-02T21:12:23.587-04:00	Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.

2021-06-02T21:13:05.597-04:00	Collecting package metadata (repodata.json): ...working... done

2021-06-02T21:17:06.653-04:00	Solving environment: ...working... done

2021-06-02T21:17:47.662-04:00	## Package Plan ## environment location: /home/ec2-user/anaconda3 added / updated specs: - conda=4.10.1

2021-06-02T21:17:47.663-04:00	The following packages will be downloaded: package | build ---------------------------|----------------- anyio-3.1.0 | py37h89c1867_0 136 KB conda-forge ca-certificates-2021.5.30 | ha878542_0 136 KB conda-forge certifi-2021.5.30 | py37h89c1867_0 141 KB conda-forge cryptography-3.4.7 | py37h5d9358c_0 1.1 MB conda-forge docutils-0.17.1 | py37h89c1867_0 762 KB conda-forge jupyter_server-1.8.0 | pyhd8ed1ab_0 255 KB conda-forge jupyterlab-3.0.16 | pyhd8ed1ab_0 5.7 MB conda-forge jupyterlab_server-2.6.0 | pyhd8ed1ab_0 44 KB conda-forge lxml-4.6.3 | py37h77fd288_0 1.5 MB conda-forge nbclassic-0.3.1 | pyhd8ed1ab_1 18 KB conda-forge urllib3-1.26.5 | pyhd8ed1ab_0 99 KB conda-forge websocket-client-0.57.0 | py37h89c1867_4 59 KB conda-forge ------------------------------------------------------------ Total: 9.9 MB

2021-06-02T21:17:47.663-04:00	The following NEW packages will be INSTALLED: anyio conda-forge/linux-64::anyio-3.1.0-py37h89c1867_0 colorama conda-forge/noarch::colorama-0.4.4-pyh9f0ad1d_0 cryptography conda-forge/linux-64::cryptography-3.4.7-py37h5d9358c_0 docutils conda-forge/linux-64::docutils-0.17.1-py37h89c1867_0 jupyter_server conda-forge/noarch::jupyter_server-1.8.0-pyhd8ed1ab_0 jupyterlab conda-forge/noarch::jupyterlab-3.0.16-pyhd8ed1ab_0 jupyterlab_server conda-forge/noarch::jupyterlab_server-2.6.0-pyhd8ed1ab_0 lxml conda-forge/linux-64::lxml-4.6.3-py37h77fd288_0 nbclassic conda-forge/noarch::nbclassic-0.3.1-pyhd8ed1ab_1 pip conda-forge/noarch::pip-21.1.2-pyhd8ed1ab_0 requests conda-forge/noarch::requests-2.25.1-pyhd3deb0d_0 sniffio conda-forge/linux-64::sniffio-1.2.0-py37h89c1867_1 urllib3 conda-forge/noarch::urllib3-1.26.5-pyhd8ed1ab_0 websocket-client conda-forge/linux-64::websocket-client-0.57.0-py37h89c1867_4

2021-06-02T21:17:47.663-04:00	The following packages will be UPDATED: ca-certificates 2020.12.5-ha878542_0 --> 2021.5.30-ha878542_0 certifi 2020.12.5-py37h89c1867_1 --> 2021.5.30-py37h89c1867_0

2021-06-02T21:17:47.663-04:00	Downloading and Extracting Packages jupyterlab_server-2. | 44 KB | | 0% jupyterlab_server-2. | 44 KB | ########## | 100% jupyterlab_server-2. | 44 KB | ########## | 100% certifi-2021.5.30 | 141 KB | | 0% certifi-2021.5.30 | 141 KB | ########## | 100% websocket-client-0.5 | 59 KB | | 0% websocket-client-0.5 | 59 KB | ########## | 100% jupyter_server-1.8.0 | 255 KB | | 0% jupyter_server-1.8.0 | 255 KB | ########## | 100% jupyter_server-1.8.0 | 255 KB | ########## | 100% nbclassic-0.3.1 | 18 KB | | 0% nbclassic-0.3.1 | 18 KB | ########## | 100%

2021-06-02T21:17:48.663-04:00	jupyterlab-3.0.16 | 5.7 MB | | 0% jupyterlab-3.0.16 | 5.7 MB | ########## | 100% jupyterlab-3.0.16 | 5.7 MB | ########## | 100%

2021-06-02T21:17:49.664-04:00	ca-certificates-2021 | 136 KB | | 0% ca-certificates-2021 | 136 KB | ########## | 100% anyio-3.1.0 | 136 KB | | 0% anyio-3.1.0 | 136 KB | ########## | 100% lxml-4.6.3 | 1.5 MB | | 0% lxml-4.6.3 | 1.5 MB | ########## | 100% lxml-4.6.3 | 1.5 MB | ########## | 100% urllib3-1.26.5 | 99 KB | | 0% urllib3-1.26.5 | 99 KB | ########## | 100% cryptography-3.4.7 | 1.1 MB | | 0% cryptography-3.4.7 | 1.1 MB | ########## | 100% cryptography-3.4.7 | 1.1 MB | ########## | 100%

2021-06-02T21:17:50.664-04:00	docutils-0.17.1 | 762 KB | | 0% docutils-0.17.1 | 762 KB | ########## | 100% docutils-0.17.1 | 762 KB | ########## | 100%

2021-06-02T21:17:50.664-04:00	Preparing transaction: ...working... done

2021-06-02T21:17:50.664-04:00	Verifying transaction: ...working... failed

2021-06-02T21:17:50.664-04:00	The environment is inconsistent, please check the package plan carefully

2021-06-02T21:17:50.664-04:00	The following packages are causing the inconsistency: - defaults/noarch::tifffile==2021.1.14=pyhd3eb1b0_1 - defaults/linux-64::numpy-base==1.19.2=py37hfa32c7d_0 - conda-forge/linux-64::numexpr==2.7.3=py37hdc94413_0 - defaults/linux-64::secretstorage==3.3.1=py37h06a4308_0 - defaults/linux-64::bokeh==2.2.3=py37_0 - defaults/linux-64::anaconda-client==1.7.2=py37_0 - defaults/linux-64::bottleneck==1.3.2=py37heb32a55_1 - defaults/linux-64::imagecodecs==2021.1.11=py37h581e88b_1 - defaults/linux-64::keyring==22.0.1=py37h06a4308_0 - defaults/noarch::dask==2021.2.0=pyhd3eb1b0_0 - defaults/linux-64::_anaconda_depends==2020.07=py37_0 - defaults/linux-64::mkl_fft==1.3.0=py37h54f3939_0 - defaults/linux-64::scikit-learn==0.23.2=py37h0573a6f_0 - defaults/linux-64::spyder==4.2.1=py37h06a4308_1 - defaults/linux-64::harfbuzz==2.4.0=hca77d97_1 - defaults/linux-64::pywavelets==1.1.1=py37h7b6447c_2 - defaults/linux-64::pytables==3.6.1=py37h71ec239_0 - defaults/noarch::anaconda-project==0.9.1=pyhd3eb1b0_1 - defaults/linux-64::scipy==1.6.2=py37h91f5cce_0 - defaults/linux-64::cairo==1.14.12=h8948797_3 - defaults/linux-64::pyopenssl==19.1.0=py37_0 - defaults/linux-64::bkcharts==0.2=py37_0 - defaults/noarch::seaborn==0.11.1=pyhd3eb1b0_0 - defaults/linux-64::pango==1.45.3=hd140c19_0 - defaults/linux-64::pandas==1.2.2=py37ha9443f7_0 - defaults/noarch::sphinx==3.5.1=pyhd3eb1b0_0 - defaults/linux-64::mkl_random==1.1.1=py37h0573a6f_0 - defaults/noarch::imageio==2.9.0=py_0 - defaults/linux-64::astropy==4.2=py37h27cfd23_0 - defaults/linux-64::conda==4.8.4=py37_0 - defaults/linux-64::matplotlib-base==3.3.4=py37h62a2d02_0 - defaults/linux-64::matplotlib==3.3.4=py37h06a4308_0 - defaults/linux-64::numba==0.51.2=py37h04863e7_1 - defaults/linux-64::scikit-image==0.17.2=py37hdf5156a_0 - defaults/linux-64::pyerfa==1.7.2=py37h27cfd23_0 - defaults/linux-64::statsmodels==0.12.2=py37h27cfd23_0 - defaults/linux-64::numpy==1.19.2=py37h54aff64_0 - defaults/linux-64::patsy==0.5.1=py37_0 - defaults/noarch::numpydoc==1.1.0=pyhd3eb1b0_1 - defaults/linux-64::h5py==2.10.0=py37h7918eee_0

2021-06-02T21:17:50.664-04:00	The environment is inconsistent, please check the package plan carefully

2021-06-02T21:17:50.665-04:00	The following packages are causing the inconsistency: - defaults/linux-64::secretstorage==3.3.1=py37h06a4308_0 - defaults/linux-64::anaconda-client==1.7.2=py37_0 - defaults/linux-64::keyring==22.0.1=py37h06a4308_0 - defaults/linux-64::_anaconda_depends==2020.07=py37_0 - defaults/linux-64::spyder==4.2.1=py37h06a4308_1 - defaults/noarch::anaconda-project==0.9.1=pyhd3eb1b0_1 - defaults/linux-64::pyopenssl==19.1.0=py37_0 - defaults/noarch::sphinx==3.5.1=pyhd3eb1b0_0 - defaults/linux-64::conda==4.8.4=py37_0 - defaults/noarch::numpydoc==1.1.0=pyhd3eb1b0_1

2021-06-02T21:17:50.665-04:00	==> WARNING: A newer version of conda exists. <== current version: 4.8.4 latest version: 4.10.1

2021-06-02T21:17:50.665-04:00	Please update conda by running $ conda update -n base -c defaults conda

2021-06-02T21:17:50.665-04:00	RemoveError: 'requests' is a dependency of conda and cannot be removed from

2021-06-02T21:17:50.665-04:00	conda's operating environment.```
2 Likes
2

Looks like I was able to fix it! Adding the line conda update --force-reinstall conda -y after echo "Updating conda" did the job. There are probably too many calls to updating conda in the script now, but it works :man_shrugging:

4 Likes
3

Hi Andrew! Thanks for the post.

I am facing the identical issue (conda install fails possibly probably due to versioning diff of the contained packages, which leads to incorrect/incomplete FastaiSageMakerStack install - indicated by no fastai kernel availability for running Jupyter notebooks).

  • I followed your advice and tried these steps w/o success:
  1. modified template for the stack directly by inserting the line below
    conda update --force-reinstall conda -y
    after
    echo “Updating conda”
  2. saved the modified template to AWS S3 bucket
  3. updated the stack from the saved template (replace template option)

This, however, hasn’t updated conda environment and hasn’t recreated the notebook instances.

  • What did I miss?
  • If you could specify the procedure you applied to fix conda/fastai kernel options availability it would be great!

Thank you

Best
PO

4

I think if you just update the CFN stack, it doesn’t re-run the creation script for the notebook. Try manually creating a notebook with the same params as the other notebook, and then it should use the new creation script.

When you create the new notebook, make sure to open and complete “Additional configuration” and “Git repository”. I think Sagemaker only charges you for running notebooks, so manually creating a new notebook shouldn’t add to your AWS charges (but also worth double checking if this is important to you).

5

Andrew, thanks for following up!

Indeed, I confirmed that simply updating FastaiSageMakerStack template doesn’t reinstall conda environment properly. The issue persists since even though the updated stack has the solution that you recommended it needs to be activated.

After some poking around I got your solution to work! Here’s the procedure I used for anyone who is facing the same issue (I assume it applies to many folks installing fastai2 course in SageMaker now):

  1. Modify the FastaiSageMakerStack template per AndrewN’s single line solution as noted above.
  2. Save the modified stack template f.e. in the AWS S3 bucket and save the address (url) of the template for the next steps.
  3. Delete the existing FastaiSageMakerStack (this wipes out all provisioned resources including the corrupt install of conda environment that caused the issue. You have the modified template stored and that’s all you need for the next step).
  4. Create a new stack from scratch using the same parameters as specified here: Amazon SageMaker | Practical Deep Learning for Coders (naming the stack FastaiSageMakerStack etc).
  5. Select the existing stack template option and enter the stack template url (generated in step 2). Create the stack.
  6. Complete Jupyter Notebooks deployment per the course instructions. The above should have installed your conda environment smoothly making fastai available as an option for Notebook kernel.

Hope this helps! Now on to the course - let’s build and train some models! :grinning:

Best
PO

3 Likes
6

Kernel_not_found
Kernel_not_found704×766 45.3 KB

Hi Peter,

I am installing fastai2 course with SageMaker. I am not getting fastai kernel option with the installation instructions. I also tried these steps mentioned (modified lines from template below), but I am seeing no change in my kernel list with these changes in the template. Can you please suggest if there is any other probable solution to this problem?

Thanks in advance.

            echo "Updating conda"
              
              conda update --force-reinstall conda -y

              conda update -n base -c defaults conda -y

              conda update --all -y
1 Like
7

Hi Shobhit!

How did you update the stack template? It is where I found the key steps to my solution. Specifically:

(1) I modified the template by editing it in the AWS CloudFormation Designer (Modifying a stack template - AWS CloudFormation).

modifying_template
modifying_template2146×1164 135 KB

(2) Then saved the modified template in S3 bucket.

(3) finally, deleted the existing stack completely and recreated in from scratch from the modified template saved in S3 bucket.

Before recreating the stack as shown above, my template modification had not fixed the errors in conda environment install as traced by the CloudWatch logs ('fastai-v4/LifecycleConfigOnCreate’ log is the one that matters):

conda_install_error
conda_install_error1987×1007 102 KB

If you have run the steps above successfully recreating the stack, conda environment should be clean resulting in making fastai kernel option available.

To check your CloudWatch logs (fastai-v4/LifecycleConfigOnCreate) follow this:
https://docs.aws.amazon.com/sagemaker/latest/dg/jupyter-logs.html

CloudWatch logs
CloudWatch logs1915×1077 127 KB

Where do you diverge from the above? I hope this helps :grinning:! If not let’s look into the issue further…

Best,
PO

3 Likes
8

Thanks Peter for the detailed explanation. With the logging, I was able to install it correctly.

To all the future users looking for this issue, the key is to wait around 25-30 minutes for the installation to finish, after the stack is created (follow the CloudWatch logs to see the finish of script before starting the notebook. How to open Logs: Follow the steps @deep_learner mentioned in previous post )

9

Shobhit

Thanks for the note! Great to know this fixing pathway has worked for you.

Best
PO

10

Hi guys!

I have faced further issues with running the Notebook instance after applying the initial solution as described above. Specifically, the notebooks are created as expected, however, after stopping the instance (fastai-v4) and coming back to work and restarting it, I would be getting a Failed (to start) error:

Notebook Instance Lifecycle Config ‘arn:aws:sagemaker:us-east-2:255007536151:notebook-instance-lifecycle-config/fastai-v4lifecycleconfig’ for Notebook Instance ‘arn:aws:sagemaker:us-east-2:255007536151:notebook-instance/fastai-v4’ took longer than 5 minutes. Please check your CloudWatch logs for more details if your Notebook Instance has Internet access.

NoteBookInstance_fails_to_start_error_marked_up
NoteBookInstance_fails_to_start_error_marked_up2075×425 37.7 KB

The error in CloudWatch OnStart log exposed that conda install inconsistency persisted if updates are allowed OnStart:

..
==> WARNING: A newer version of conda exists. <==
  current version: 4.8.4
  latest version: 4.10.1
Please update conda by running

    $ conda update -n base -c defaults conda

After further investigation, I wound up editing the FastaiSageMakerStack template (sagemaker-cfn-course-v4.yml) by removing updating conda section in OnStart script section:

templateURL = https://fastai-cfn.s3.amazonaws.com/sagemaker-cfn-course-v4.yml
stackName = FastaiSageMakerStack

This results in updating conda env only after the initial creation (OnCreate) of the Notebook instance (fastai-v4) and not attempting to update it on subsequent starts (OnStart). While this appears to ‘freeze’ the installed conda env version, it has solved the issue.

Here is the code change applied to OnStart section of the template:

original ver (modified per above posts):

             echo "Updating conda"
              conda update --force-reinstall conda -y
              conda update -n base -c defaults conda -y
              conda update --all -y

updated version (last three lines removed in OnStart section):

             echo "Updating conda - skipped in FastaiSageMakerStack template v6 (sagemaker-cfn-course-v6.yml)"

Below is the updated ver (sagemaker-cfn-course-v4_updated.yml) of the stack template to install the stack from scratch instead of creating the stack from the templates listed here: Amazon SageMaker | Practical Deep Learning for Coders (which results in the error).

To create the stack from scratch, save the template below as sagemaker-cfn-course-v4_updated.yml file. Then delete the failing stack and create it from scratch from this template.

Checking the CloudWatch logs (OnCreate and OnStart) should reflect now errors and successful avoiding of attempting to update conda OnStart

Parameters:
  InstanceType:
    Type: String
    Default: ml.p2.xlarge
    AllowedValues:
      - ml.p3.2xlarge
      - ml.p2.xlarge
    Description: Enter the SageMaker Notebook instance type
  VolumeSize:
    Type: Number
    Default: 50
    Description: Enter the size of the EBS volume attached to the notebook instance
    MaxValue: 17592
    MinValue: 5
Resources:
  Fastai2SagemakerNotebookfastaiv4NotebookRoleA75B4C74:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: sagemaker.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/AmazonSageMakerFullAccess
    Metadata:
      aws:cdk:path: CdkFastaiv2SagemakerNbStack/Fastai2SagemakerNotebook/fastai-v4NotebookRole/Resource
  Fastai2SagemakerNotebookfastaiv4LifecycleConfigD72E2247:
    Type: AWS::SageMaker::NotebookInstanceLifecycleConfig
    Properties:
      NotebookInstanceLifecycleConfigName: fastai-v4LifecycleConfig
      OnCreate:
        - Content:
            Fn::Base64: >-
              #!/bin/bash


              set -e


              echo "Starting on Create script"


              sudo -i -u ec2-user bash <<EOF

              touch /home/ec2-user/SageMaker/.create-notebook

              EOF


              cat > /home/ec2-user/SageMaker/.fastai-install.sh <<\EOF

              #!/bin/bash

              set -e

              echo "Creating dirs and symlinks"

              mkdir -p /home/ec2-user/SageMaker/.cache

              mkdir -p /home/ec2-user/SageMaker/.fastai

              [ ! -L "/home/ec2-user/.cache" ] && ln -s /home/ec2-user/SageMaker/.cache /home/ec2-user/.cache

              [ ! -L "/home/ec2-user/.fastai" ] && ln -s /home/ec2-user/SageMaker/.fastai /home/ec2-user/.fastai


              echo "Updating conda"

              conda update --force-reinstall conda -y
            
              conda update -n base -c defaults conda -y

              conda update --all -y

              echo "Starting conda create command for fastai env"

              conda create -mqyp /home/ec2-user/SageMaker/.env/fastai python=3.6

              echo "Activate fastai conda env"

              conda init bash

              source ~/.bashrc

              conda activate /home/ec2-user/SageMaker/.env/fastai

              echo "Install ipython kernel and widgets"

              conda install ipywidgets ipykernel -y

              echo "Installing fastai lib"

              pip install -r /home/ec2-user/SageMaker/fastbook/requirements.txt

              pip install fastbook sagemaker

              echo "Installing Jupyter kernel for fastai"

              python -m ipykernel install --name 'fastai' --user

              echo "Finished installing fastai conda env"

              echo "Install Jupyter nbextensions"

              conda activate JupyterSystemEnv

              pip install jupyter_contrib_nbextensions

              jupyter contrib nbextensions install --user

              echo "Restarting jupyter notebook server"

              pkill -f jupyter-notebook

              rm /home/ec2-user/SageMaker/.create-notebook

              echo "Exiting install script"

              EOF


              chown ec2-user:ec2-user /home/ec2-user/SageMaker/.fastai-install.sh

              chmod 755 /home/ec2-user/SageMaker/.fastai-install.sh


              sudo -i -u ec2-user bash <<EOF

              nohup /home/ec2-user/SageMaker/.fastai-install.sh &

              EOF


              echo "Finishing on Create script"
      OnStart:
        - Content:
            Fn::Base64: >-
              #!/bin/bash


              set -e


              echo "Starting on Start script"


              sudo -i -u ec2-user bash << EOF

              if [[ -f /home/ec2-user/SageMaker/.create-notebook ]]; then
                  echo "Skipping as currently installing conda env"
              else
                  # create symlinks to EBS volume
                  echo "Creating symlinks"
                  ln -s /home/ec2-user/SageMaker/.fastai /home/ec2-user/.fastai
                  echo "Updating conda - skipped in FastaiSageMakerStack template v4_updated (sagemaker-cfn-course-v4_updated.yml)"
                  echo "Activate fastai conda env"
                  conda init bash
                  source ~/.bashrc
                  conda activate /home/ec2-user/SageMaker/.env/fastai
                  echo "Updating fastai packages"
                  pip install fastai fastcore sagemaker --upgrade
                  echo "Installing Jupyter kernel"
                  python -m ipykernel install --name 'fastai' --user
                  echo "Install Jupyter nbextensions"
                  conda activate JupyterSystemEnv
                  pip install jupyter_contrib_nbextensions
                  jupyter contrib nbextensions install --user
                  echo "Restarting jupyter notebook server"
                  pkill -f jupyter-notebook
                  echo "Finished setting up Jupyter kernel"
              fi

              EOF


              echo "Finishing on Start script"
    Metadata:
      aws:cdk:path: CdkFastaiv2SagemakerNbStack/Fastai2SagemakerNotebook/fastai-v4LifecycleConfig
  Fastai2SagemakerNotebookfastaiv4NotebookInstance7C46E7E0:
    Type: AWS::SageMaker::NotebookInstance
    Properties:
      InstanceType:
        Ref: InstanceType
      RoleArn:
        Fn::GetAtt:
          - Fastai2SagemakerNotebookfastaiv4NotebookRoleA75B4C74
          - Arn
      DefaultCodeRepository: https://github.com/fastai/fastbook
      LifecycleConfigName: fastai-v4LifecycleConfig
      NotebookInstanceName: fastai-v4
      VolumeSizeInGB:
        Ref: VolumeSize
    Metadata:
      aws:cdk:path: CdkFastaiv2SagemakerNbStack/Fastai2SagemakerNotebook/fastai-v4NotebookInstance
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Modules: aws-cdk=1.60.0,@aws-cdk/aws-iam=1.60.0,@aws-cdk/aws-sagemaker=1.60.0,@aws-cdk/cloud-assembly-schema=1.60.0,@aws-cdk/core=1.60.0,@aws-cdk/cx-api=1.60.0,@aws-cdk/region-info=1.60.0,jsii-runtime=node.js/v14.8.0
    Condition: CDKMetadataAvailable
Conditions:
  CDKMetadataAvailable:
    Fn::Or:
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ca-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-northwest-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-central-1
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-2
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-3
          - Fn::Equals:
              - Ref: AWS::Region
              - me-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - sa-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-2
          - Fn::Equals:
              - Ref: AWS::Region
              - us-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-west-2

I hope this helps.

Best,
PO

11

Hi Andrew!

It would be great if you shared how exactly you applied the fix to conda install packages inconsistency. As documented above, I faced a deeper issues OnStart and wound up removing conda updates from it (essentially, freezing the installed conda ver). How did you apply the solution that worked for you?

Thank you

Best
PO

12

Hi Shobhit!

I would appreciate if you could share how exactly you applied the fix i.e. did you wind up with a similar approach that I posted above (removing conda updates from OnStart script)? If not what has worked for you?

Thank you

Best
PO

13

Looks like you got it figured out, but I only ever modified the onCreate script with that single line from my original solution post.

14

Thanks, Andrew.

Right, that seems to be the right fix.

Best
PO

15

Hi Peter,

Sorry for the late response. I also faced the similar issue of failing when restarting the notebook instance. I ended up experimenting and moving to google co-lab. I believe you figured it out with the change it in the OnStart script. I hope it is working well.

Regards,
Shobhit

16

Hi Shobhit

Thanks for the note! The workaround as posted above does fix the issue. If you want to give AWS SageMaker another try, you can try this fix it takes only half hour to install a clean FastaiSageMakerStack (I waited for an hour to be safe) :grinning:.

Best
PO

17

Hi all

FYI
The issue should now be fixed via script change/re-pointing here:

@AndrewN and @shobhit: thank you for getting to the bottom of the issue!

Best
PO