For those of us that want to run on AWS

Part 1 (2018)
1

@jeremy,

Sorry to add yet another post to the sea of existing AWS related posts, but as someone who really wants to learn and understand how to work on AWS (esp. as it pertains to machine learning development), I have a few questions:

  1. Should we start with a vanilla Ubuntu AMI or use one of the Deep Learning AMIs (see here for one that looks like it meets the specs of the course)?

  2. If the recommendation is to go with the plain vanilla instance, is there any modifications you would recommend to the install script @radek shared here.

  3. What is the minimum and recommended about of storage to use?

I know that you recommend using crestle or paperspace for the time being because it is easier to get up and running, but I just feel like configuring AWS is part of the learning experience since this is where most companies will be doing their ML work.

Thanks much!

2

If you are starting with AWS, it might be better to use an Deep Learning AMI. I believe Jeremy plans to create one for the course. AWS also has an official Deep Learning AMI. You could start with that and install anything else you need like updating PyTorch etc (https://aws.amazon.com/amazon-ai/amis/),

I use a basic Ubuntu 16.04 AMI and then use nvidia-docker to run my deep learning tasks. You can use Pytorch or TensorFlow Docker Files (https://github.com/pytorch/pytorch/blob/master/Dockerfile / https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/docker/Dockerfile.gpu). There’s lot of similarity between them. I like the one from PyTorch because it uses Conda. Tensorflow one uses Pip. Use the one that you prefer.

I will detail down my setup at some point and share it here.

5 Likes
3

good stuff @ramesh.

I’d be really interested to hear about the docker approach/configuration from beginning to end. I don’t run Docker on my linux laptop, but I do on Windows. Definitely seems like a good idea for folks regardless of what platform they are running on.

-wg

1 Like