How to build an AWS EC2 environment for Python 3, Tensorflow, and Keras 2?

I don’t have the cash to build my own machine and that the weakest part for me is the proper build and use of an AWS EC2 instance for machine learning.

I really want to learn how to build an appropriate instance from scratch in order to gain a real understanding of what is going on … and I want to implement an instance that will support using Jupyter Notebooks against a Python 3, TF, Keras 2 technology stack in order to do the Kaggle competitions in this course.

So … can anyone recommend some good tutorials or walk-thrus?

The necessary steps are listed in the install-gpu.sh script - if you read through that, and update the steps for the versions you list, you’ll be up and running!

Cool thanks Jeremy!

Any thoughts on using the new AWS Deep Learning AMIs as a starting point?

I see some discussion on folks going to Google Cloud but as I have $500 in credits for AWS … I’m going to stick with it and just want to setup a decent enough instance.

Nice! Where did you get the credits?

I wasn’t a big fan of the AMIs last time I looked.

Thru attending an event at UCSD. Was a pleasant surprise after the fact :slight_smile:

Any particular reason?

Btw, how much storage do you have configured for the default instance the course uses?

Just an FYI … this worked flawlessly (with one minor update to set LD_LIBRARY_PATH env. variable per the tensorflow installation documentation)

If there is a place to share my modifications outside of the forums I’d be glad too … just need to know where.

Thanks for your help!

Is there any equivalent setup script for Windows-based computers already built?

UPDATE: I tried and mostly reached a working state. However, CUDA fails to compile any GPU code, because it requires Microsoft Visual Studio. That would not be a problem because I have it installed, but I believe it conflicts with the compiler used to build Anaconda, which seems to not be MSVC, rather gcc or gcc/mingw. There doesn’t seem to be any solution to this, and CUDA only supports MSVC on Windows. I will try to just disable GPU compilation, but there goes the whole point of trying to use my Windows workstation… :disappointed:

It definitely works - I’ve had it working as have others on the forum. Search the forums and you’ll find some threads that should handle it.

1 Like

Thanks :slight_smile: I’ve just added all regular forum users (anyone with >=1 like and >=2 posts) to the wiki and sent them an email with the login details. So that would be a great place to share your modifications!

(Sorry it’s taken me so long to get onto creating wiki logins!..)

1 Like

Have you searched the forums on running the notebooks on a Windows machine?

I don’t have the links handy but I was able to piecemeal things together to get things running on my XPS 15 notebook using the 960 GPU. It was a tad painful but that is mostly because it was new to me. Definitely recommend going with a Linux build if possible, but Windows is doable. In fact, my development process right now is to do all my work on my PC with a sample dataset, check in everything to github … launch my EC2 instance, download my code from github, and run everything against the full datasets. Still learning and thinking about best approach to developing deep learning solutions … but this is working for me.

If you haven’t found the Windows links yet … lmk and I’ll dig them up.

-wg

Found this which seems to be quite thorough, https://github.com/Theano/Theano/issues/5348.
I’ll try tonight. If you can find forum posts, please let me know.

At this point it’s a matter of principle :slight_smile: so I want to have this working.
Running on AWS it’s much easier and faster, and I can see why it’s recommended though.

1 Like

Here is the thread I was thinking of … lots of material in here to help get you up and running on a Windows PC.

2 Likes