Sagemaker Notebook deployment problem - no FastAi kernel

I’m deploying to Sagemaker through these instructions to a p3 instance. The notebook is created fine but when I’m prompted for a kernel, there is no fastai option.

I’ve tried using the fix at the end of this thread.

Here are the logs from my LifeCycleConfigOnCreate. Will keep trying to debug, but any help is appreciated!

2021-06-02T21:07:33.523-04:00	Starting on Create script

2021-06-02T21:07:34.523-04:00	Finishing on Create script

2021-06-02T21:07:34.523-04:00	Creating dirs and symlinks

2021-06-02T21:07:34.523-04:00	Updating conda

2021-06-02T21:07:59.529-04:00	Collecting package metadata (current_repodata.json): ...working... done

2021-06-02T21:09:34.549-04:00	Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.

2021-06-02T21:12:23.587-04:00	Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.

2021-06-02T21:13:05.597-04:00	Collecting package metadata (repodata.json): ...working... done

2021-06-02T21:17:06.653-04:00	Solving environment: ...working... done

2021-06-02T21:17:47.662-04:00	## Package Plan ## environment location: /home/ec2-user/anaconda3 added / updated specs: - conda=4.10.1

2021-06-02T21:17:47.663-04:00	The following packages will be downloaded: package | build ---------------------------|----------------- anyio-3.1.0 | py37h89c1867_0 136 KB conda-forge ca-certificates-2021.5.30 | ha878542_0 136 KB conda-forge certifi-2021.5.30 | py37h89c1867_0 141 KB conda-forge cryptography-3.4.7 | py37h5d9358c_0 1.1 MB conda-forge docutils-0.17.1 | py37h89c1867_0 762 KB conda-forge jupyter_server-1.8.0 | pyhd8ed1ab_0 255 KB conda-forge jupyterlab-3.0.16 | pyhd8ed1ab_0 5.7 MB conda-forge jupyterlab_server-2.6.0 | pyhd8ed1ab_0 44 KB conda-forge lxml-4.6.3 | py37h77fd288_0 1.5 MB conda-forge nbclassic-0.3.1 | pyhd8ed1ab_1 18 KB conda-forge urllib3-1.26.5 | pyhd8ed1ab_0 99 KB conda-forge websocket-client-0.57.0 | py37h89c1867_4 59 KB conda-forge ------------------------------------------------------------ Total: 9.9 MB

2021-06-02T21:17:47.663-04:00	The following NEW packages will be INSTALLED: anyio conda-forge/linux-64::anyio-3.1.0-py37h89c1867_0 colorama conda-forge/noarch::colorama-0.4.4-pyh9f0ad1d_0 cryptography conda-forge/linux-64::cryptography-3.4.7-py37h5d9358c_0 docutils conda-forge/linux-64::docutils-0.17.1-py37h89c1867_0 jupyter_server conda-forge/noarch::jupyter_server-1.8.0-pyhd8ed1ab_0 jupyterlab conda-forge/noarch::jupyterlab-3.0.16-pyhd8ed1ab_0 jupyterlab_server conda-forge/noarch::jupyterlab_server-2.6.0-pyhd8ed1ab_0 lxml conda-forge/linux-64::lxml-4.6.3-py37h77fd288_0 nbclassic conda-forge/noarch::nbclassic-0.3.1-pyhd8ed1ab_1 pip conda-forge/noarch::pip-21.1.2-pyhd8ed1ab_0 requests conda-forge/noarch::requests-2.25.1-pyhd3deb0d_0 sniffio conda-forge/linux-64::sniffio-1.2.0-py37h89c1867_1 urllib3 conda-forge/noarch::urllib3-1.26.5-pyhd8ed1ab_0 websocket-client conda-forge/linux-64::websocket-client-0.57.0-py37h89c1867_4

2021-06-02T21:17:47.663-04:00	The following packages will be UPDATED: ca-certificates 2020.12.5-ha878542_0 --> 2021.5.30-ha878542_0 certifi 2020.12.5-py37h89c1867_1 --> 2021.5.30-py37h89c1867_0

2021-06-02T21:17:47.663-04:00	Downloading and Extracting Packages jupyterlab_server-2. | 44 KB | | 0% jupyterlab_server-2. | 44 KB | ########## | 100% jupyterlab_server-2. | 44 KB | ########## | 100% certifi-2021.5.30 | 141 KB | | 0% certifi-2021.5.30 | 141 KB | ########## | 100% websocket-client-0.5 | 59 KB | | 0% websocket-client-0.5 | 59 KB | ########## | 100% jupyter_server-1.8.0 | 255 KB | | 0% jupyter_server-1.8.0 | 255 KB | ########## | 100% jupyter_server-1.8.0 | 255 KB | ########## | 100% nbclassic-0.3.1 | 18 KB | | 0% nbclassic-0.3.1 | 18 KB | ########## | 100%

2021-06-02T21:17:48.663-04:00	jupyterlab-3.0.16 | 5.7 MB | | 0% jupyterlab-3.0.16 | 5.7 MB | ########## | 100% jupyterlab-3.0.16 | 5.7 MB | ########## | 100%

2021-06-02T21:17:49.664-04:00	ca-certificates-2021 | 136 KB | | 0% ca-certificates-2021 | 136 KB | ########## | 100% anyio-3.1.0 | 136 KB | | 0% anyio-3.1.0 | 136 KB | ########## | 100% lxml-4.6.3 | 1.5 MB | | 0% lxml-4.6.3 | 1.5 MB | ########## | 100% lxml-4.6.3 | 1.5 MB | ########## | 100% urllib3-1.26.5 | 99 KB | | 0% urllib3-1.26.5 | 99 KB | ########## | 100% cryptography-3.4.7 | 1.1 MB | | 0% cryptography-3.4.7 | 1.1 MB | ########## | 100% cryptography-3.4.7 | 1.1 MB | ########## | 100%

2021-06-02T21:17:50.664-04:00	docutils-0.17.1 | 762 KB | | 0% docutils-0.17.1 | 762 KB | ########## | 100% docutils-0.17.1 | 762 KB | ########## | 100%

2021-06-02T21:17:50.664-04:00	Preparing transaction: ...working... done

2021-06-02T21:17:50.664-04:00	Verifying transaction: ...working... failed

2021-06-02T21:17:50.664-04:00	The environment is inconsistent, please check the package plan carefully

2021-06-02T21:17:50.664-04:00	The following packages are causing the inconsistency: - defaults/noarch::tifffile==2021.1.14=pyhd3eb1b0_1 - defaults/linux-64::numpy-base==1.19.2=py37hfa32c7d_0 - conda-forge/linux-64::numexpr==2.7.3=py37hdc94413_0 - defaults/linux-64::secretstorage==3.3.1=py37h06a4308_0 - defaults/linux-64::bokeh==2.2.3=py37_0 - defaults/linux-64::anaconda-client==1.7.2=py37_0 - defaults/linux-64::bottleneck==1.3.2=py37heb32a55_1 - defaults/linux-64::imagecodecs==2021.1.11=py37h581e88b_1 - defaults/linux-64::keyring==22.0.1=py37h06a4308_0 - defaults/noarch::dask==2021.2.0=pyhd3eb1b0_0 - defaults/linux-64::_anaconda_depends==2020.07=py37_0 - defaults/linux-64::mkl_fft==1.3.0=py37h54f3939_0 - defaults/linux-64::scikit-learn==0.23.2=py37h0573a6f_0 - defaults/linux-64::spyder==4.2.1=py37h06a4308_1 - defaults/linux-64::harfbuzz==2.4.0=hca77d97_1 - defaults/linux-64::pywavelets==1.1.1=py37h7b6447c_2 - defaults/linux-64::pytables==3.6.1=py37h71ec239_0 - defaults/noarch::anaconda-project==0.9.1=pyhd3eb1b0_1 - defaults/linux-64::scipy==1.6.2=py37h91f5cce_0 - defaults/linux-64::cairo==1.14.12=h8948797_3 - defaults/linux-64::pyopenssl==19.1.0=py37_0 - defaults/linux-64::bkcharts==0.2=py37_0 - defaults/noarch::seaborn==0.11.1=pyhd3eb1b0_0 - defaults/linux-64::pango==1.45.3=hd140c19_0 - defaults/linux-64::pandas==1.2.2=py37ha9443f7_0 - defaults/noarch::sphinx==3.5.1=pyhd3eb1b0_0 - defaults/linux-64::mkl_random==1.1.1=py37h0573a6f_0 - defaults/noarch::imageio==2.9.0=py_0 - defaults/linux-64::astropy==4.2=py37h27cfd23_0 - defaults/linux-64::conda==4.8.4=py37_0 - defaults/linux-64::matplotlib-base==3.3.4=py37h62a2d02_0 - defaults/linux-64::matplotlib==3.3.4=py37h06a4308_0 - defaults/linux-64::numba==0.51.2=py37h04863e7_1 - defaults/linux-64::scikit-image==0.17.2=py37hdf5156a_0 - defaults/linux-64::pyerfa==1.7.2=py37h27cfd23_0 - defaults/linux-64::statsmodels==0.12.2=py37h27cfd23_0 - defaults/linux-64::numpy==1.19.2=py37h54aff64_0 - defaults/linux-64::patsy==0.5.1=py37_0 - defaults/noarch::numpydoc==1.1.0=pyhd3eb1b0_1 - defaults/linux-64::h5py==2.10.0=py37h7918eee_0

2021-06-02T21:17:50.664-04:00	The environment is inconsistent, please check the package plan carefully

2021-06-02T21:17:50.665-04:00	The following packages are causing the inconsistency: - defaults/linux-64::secretstorage==3.3.1=py37h06a4308_0 - defaults/linux-64::anaconda-client==1.7.2=py37_0 - defaults/linux-64::keyring==22.0.1=py37h06a4308_0 - defaults/linux-64::_anaconda_depends==2020.07=py37_0 - defaults/linux-64::spyder==4.2.1=py37h06a4308_1 - defaults/noarch::anaconda-project==0.9.1=pyhd3eb1b0_1 - defaults/linux-64::pyopenssl==19.1.0=py37_0 - defaults/noarch::sphinx==3.5.1=pyhd3eb1b0_0 - defaults/linux-64::conda==4.8.4=py37_0 - defaults/noarch::numpydoc==1.1.0=pyhd3eb1b0_1

2021-06-02T21:17:50.665-04:00	==> WARNING: A newer version of conda exists. <== current version: 4.8.4 latest version: 4.10.1

2021-06-02T21:17:50.665-04:00	Please update conda by running $ conda update -n base -c defaults conda

2021-06-02T21:17:50.665-04:00	RemoveError: 'requests' is a dependency of conda and cannot be removed from

2021-06-02T21:17:50.665-04:00	conda's operating environment.```
1 Like

Looks like I was able to fix it! Adding the line conda update --force-reinstall conda -y after echo "Updating conda" did the job. There are probably too many calls to updating conda in the script now, but it works :man_shrugging:

1 Like

Hi Andrew! Thanks for the post.

I am facing the identical issue (conda install fails possibly probably due to versioning diff of the contained packages, which leads to incorrect/incomplete FastaiSageMakerStack install - indicated by no fastai kernel availability for running Jupyter notebooks).

  • I followed your advice and tried these steps w/o success:
  1. modified template for the stack directly by inserting the line below
    conda update --force-reinstall conda -y
    after
    echo “Updating conda”
  2. saved the modified template to AWS S3 bucket
  3. updated the stack from the saved template (replace template option)

This, however, hasn’t updated conda environment and hasn’t recreated the notebook instances.

  • What did I miss?
  • If you could specify the procedure you applied to fix conda/fastai kernel options availability it would be great!

Thank you

Best
PO

I think if you just update the CFN stack, it doesn’t re-run the creation script for the notebook. Try manually creating a notebook with the same params as the other notebook, and then it should use the new creation script.

When you create the new notebook, make sure to open and complete “Additional configuration” and “Git repository”. I think Sagemaker only charges you for running notebooks, so manually creating a new notebook shouldn’t add to your AWS charges (but also worth double checking if this is important to you).

Andrew, thanks for following up!

Indeed, I confirmed that simply updating FastaiSageMakerStack template doesn’t reinstall conda environment properly. The issue persists since even though the updated stack has the solution that you recommended it needs to be activated.

After some poking around I got your solution to work! Here’s the procedure I used for anyone who is facing the same issue (I assume it applies to many folks installing fastai2 course in SageMaker now):

  1. Modify the FastaiSageMakerStack template per AndrewN’s single line solution as noted above.
  2. Save the modified stack template f.e. in the AWS S3 bucket and save the address (url) of the template for the next steps.
  3. Delete the existing FastaiSageMakerStack (this wipes out all provisioned resources including the corrupt install of conda environment that caused the issue. You have the modified template stored and that’s all you need for the next step).
  4. Create a new stack from scratch using the same parameters as specified here: Amazon SageMaker | Practical Deep Learning for Coders (naming the stack FastaiSageMakerStack etc).
  5. Select the existing stack template option and enter the stack template url (generated in step 2). Create the stack.
  6. Complete Jupyter Notebooks deployment per the course instructions. The above should have installed your conda environment smoothly making fastai available as an option for Notebook kernel.

Hope this helps! Now on to the course - let’s build and train some models! :grinning:

Best
PO

1 Like


Hi Peter,

I am installing fastai2 course with SageMaker. I am not getting fastai kernel option with the installation instructions. I also tried these steps mentioned (modified lines from template below), but I am seeing no change in my kernel list with these changes in the template. Can you please suggest if there is any other probable solution to this problem?

Thanks in advance.

            echo "Updating conda"
              
              conda update --force-reinstall conda -y

              conda update -n base -c defaults conda -y

              conda update --all -y

Hi Shobhit!

How did you update the stack template? It is where I found the key steps to my solution. Specifically:

(1) I modified the template by editing it in the AWS CloudFormation Designer (Modifying a stack template - AWS CloudFormation).

(2) Then saved the modified template in S3 bucket.

(3) finally, deleted the existing stack completely and recreated in from scratch from the modified template saved in S3 bucket.

Before recreating the stack as shown above, my template modification had not fixed the errors in conda environment install as traced by the CloudWatch logs ('fastai-v4/LifecycleConfigOnCreate’ log is the one that matters):

If you have run the steps above successfully recreating the stack, conda environment should be clean resulting in making fastai kernel option available.

To check your CloudWatch logs (fastai-v4/LifecycleConfigOnCreate) follow this:
https://docs.aws.amazon.com/sagemaker/latest/dg/jupyter-logs.html

Where do you diverge from the above? I hope this helps :grinning:! If not let’s look into the issue further…

Best,
PO