Platform: AWS EC2 (DLAMI) ✅

Part 1 (2019)
41

Not sure if this is EC2 specific, but after following the usual “going back to work” instructions after a month, I get the following error when I try to run Jupyter notebook:

$ jupyter notebook
$ [C 22:48:54.046 NotebookApp] Bad config encountered during initialization:
$ [C 22:48:54.047 NotebookApp] The 'kernel_spec_manager_class' trait of <notebook.notebookapp.NotebookApp object at 0x7f0823192cf8> instance must be a type, but 'environment_kernels.EnvironmentKernelSpecManager' could not be imported

Note that this is for the fast.ai AWS image used for the class, and not from a new Ubuntu 16/18 EC2 instance.

42

I created an instance using 'Deep Learning AMI (Ubuntu) Version 21.0 ’ as I couldn’t find the option for ‘Deep Learning AMI (Ubuntu) Version 16.0’.

But I am getting the crash problem above 'The kernel appears to have died…
when running learn = create_cnn(…)

Can you please let me know if this is the correct AMI or if I should use another?

Many thanks

Jonathan

image

1 Like
43

This approach worked for me a few days ago, but I had to start over from scratch and now the kernel is dying as soon as I create a ConvLearner in notebook 1.

44

These instructions seem to be working: https://github.com/krishnakalyan3/course-v3/blob/aed64af19b34bcf0ddf1263bfd7d0e1744aac884/docs/start_aws.md

4 Likes
45

That worked for me. Couldn’t get other AWS EC2’s working

1 Like
46

I created a p2.xlarge then ran the commands

On ‘sudo apt-get install cuda’ it unpacks cuda files

but I am getting errors like the following…

Unpacking cuda-libraries-10-0 (10.0.130-1) …
dpkg: unrecoverable fatal error, aborting:
unable to flush /var/lib/dpkg/updates/tmp.i after padding: No space left on device
E: Sub-process /usr/bin/dpkg returned an error code (2)

ubuntu@ip-10-0-0-66:~$ nvidia-smi
NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

47

Looks like DLAMI is having lot of problems, I installed as per instructions somehow fastai library could not be properly installed and I keep getting errors like undefined for example on the untar_data method. Of course I follwed the FAQ guide to uninstall them but no success. Moving to easier crestle.ai option.

1 Like
48

The installation of libraries failed because the hard drive (EBS) is too small (the default it 8GB).

When setting up you should change the size (under volumes in the “Review” page) to something bigger to handle the libraries and the data (I used 75GB).

Now you should be able to resize the volume (but you might need to repartition the hard drive) or add a new volume, but if you’re not too attached to the instance it might be easier to start over.

49

In the course, Ubuntu AMI is used. I noticed there is also an Amazon linux type of AMI and I can run lesson 1 with it no problem.

Is there any difference between the Ubuntu AMI and Amazon Linux AMI?

Any suggestions?

50

Just try to bump the above question.

Thanks :grinning:

51

I am getting conda update conda error…

Pasting below the error :

Blockquote
ubuntu@ip-172-31-30-171:~$ conda update conda

CorruptedEnvironmentError: The target environment has been corrupted. Corrupted environments most commonly
occur when the conda process is force-terminated while in an unlink-link
transaction.
environment location: /home/ubuntu/anaconda3
corrupted file: /home/ubuntu/anaconda3/conda-meta/fastai-1.0.50.post1-1.json

Blockquote

Is there anything which can be done to solve the error

52

I followed the current instructions for aws exactly and tried both DLAMI Ver 16 and current (23.1). When I get to the step to run jupyter notebook I get the following error in the ssh-conntected terminal:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/bin/jupyter", line 7, in <module>
    from jupyter_core.command import main

I tried googling this for a few hours but to no avail. Any help would be much appreciated.
ModuleNotFoundError: No module named ‘jupyter_core’

1 Like
53

@davidpfahler Was this fixed? I am in the same situation. I tried conda install -c anaconda jupyter_core, still it fails due to environment inconsistency.

54

I still have the same issue. I will post, if I get it to work.

55

Seems like I got it to work. Before conda install -c pytorch -c fastai fastai pytorch torchvision cuda92 I did the following:

conda init bash
source ~/.bashrc
conda activate pytorch_p36

That seems to to the trick. I am working a jupyter notebook right now.

3 Likes
56

Thanks. That worked!

57

Let me first say that I am a newbie using anaconda and EC2-instances.

I ran into what I figure is a similar problem as the one you describe. Also tried your solution (conda init bash etc) which at least allowed me to fire up Jupyter. But my AWS-instance is still littered with problems which didn’t exist two weeks ago.

The major issue seems to be some sort of environment inconsistencies. I am however a noob, so I have no idea if that’s the case. At the moment I am trying to install the relevant packages for unpacking the tar files in Lesson 3’s Kaggle satellite images.

This input (run in the notebook):

! conda install --yes --prefix {sys.prefix} -c haasad eidl7zip

crashes and produces the following error message:

Collecting package metadata (current_repodata.json): done
Solving environment: done
WARNING conda.core.package_cache_data:_make_single_record(350): Encountered corrupt package tarball at /home/ubuntu/anaconda3/pkgs/_libgcc_mutex-0.1-main.conda. Conda has removed it, but you need to re-run conda to download it again.
WARNING conda.core.package_cache_data:_make_single_record(350): Encountered corrupt package tarball at /home/ubuntu/anaconda3/pkgs/ca-certificates-2019.5.15-0.conda. Conda has removed it, but you need to re-run conda to download it again.
WARNING conda.core.package_cache_data:_make_single_record(350): Encountered corrupt package tarball at /home/ubuntu/anaconda3/pkgs/certifi-2019.6.16-py36_0.conda. Conda has removed it, but you need to re-run conda to download it again.
WARNING conda.core.package_cache_data:_make_single_record(350): Encountered corrupt package tarball at /home/ubuntu/anaconda3/pkgs/openssl-1.0.2s-h7b6447c_0.conda. Conda has removed it, but you need to re-run conda to download it again.

Package Plan

environment location: /home/ubuntu/anaconda3/envs/pytorch_p36

added / updated specs:
- eidl7zip

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
_libgcc_mutex-0.1          |             main           3 KB
ca-certificates-2019.5.15  |                0         126 KB
certifi-2019.6.16          |           py36_0         150 KB
openssl-1.0.2s             |       h7b6447c_0         2.1 MB
------------------------------------------------------------
                                       Total:         2.4 MB

The following NEW packages will be INSTALLED:

_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
eidl7zip haasad/linux-64::eidl7zip-1.0.0-1

The following packages will be UPDATED:

ca-certificates 2019.1.23-0 --> 2019.5.15-0
certifi 2019.3.9-py36_0 --> 2019.6.16-py36_0
openssl 1.0.2r-h7b6447c_0 --> 1.0.2s-h7b6447c_0

Downloading and Extracting Packages
ca-certificates-2019 | 126 KB | ##################################### | 100%
certifi-2019.6.16 | 150 KB | ##################################### | 100%
_libgcc_mutex-0.1 | 3 KB | ##################################### | 100%
openssl-1.0.2s | 2.1 MB | ##################################### | 100%

InvalidArchiveError(‘Error with archive /home/ubuntu/anaconda3/pkgs/ca-certificates-2019.5.15-0.conda. You probably need to delete and re-download or re-create this file. Message from libarchive was:\n\nChild process exited with status 127 (errno=-1, retcode=-30, archive_p=94305117464384)’)
InvalidArchiveError(‘Error with archive /home/ubuntu/anaconda3/pkgs/certifi-2019.6.16-py36_0.conda. You probably need to delete and re-download or re-create this file. Message from libarchive was:\n\nChild process exited with status 127 (errno=-1, retcode=-30, archive_p=94305117464384)’)
InvalidArchiveError(‘Error with archive /home/ubuntu/anaconda3/pkgs/_libgcc_mutex-0.1-main.conda. You probably need to delete and re-download or re-create this file. Message from libarchive was:\n\nChild process exited with status 127 (errno=-1, retcode=-30, archive_p=94305117464384)’)
InvalidArchiveError(‘Error with archive /home/ubuntu/anaconda3/pkgs/openssl-1.0.2s-h7b6447c_0.conda. You probably need to delete and re-download or re-create this file. Message from libarchive was:\n\nChild process exited with status 127 (errno=-1, retcode=-30, archive_p=94305117464384)’)

Anyone had similar issues? Any solutions at hand? If it is any clue to the problem, I am also getting channel 3: open failed: connect failed: Connection refused in my instance terminal. ¯_(ツ)_/¯

58

Thanks. This works .

59

Hi Johan,

welcome the fastai forums! If you are just getting started with the course or with fastai in general, aws might not be the best choice. For just following the course, Google colab has worked well for me and is free!

However, I do not want to discourage you from tackling the issue you have if you are trying to get fastai working with aws, if you want to do that. I don’t think I can help with your exact error. What I would recommend is to delete the instance and start from scratch. Then follow the guide, but use the latest version of the Deep Learning AMI and inject the steps I described above.

Let me know if that still doesn’t get you going.

60

Thanks for taking your time @davidpfahler,

I have tried creating new instances several times, but no luck. My latest try I did get other issues however. This time it seemed to be version inconsistencies similar to the ones posted here. The solution presented in that thread (conda remove pykerberos) didn’t solve my issue unfortunatly :confused:

I’m gonna wait a week or two and then try again with a new instance. If the same issues arise, and I still can’t find any solution on the forum, I am going to try a different service than AWS.

Many thanks, again, for your reply!