Howto: Automated AWS spot instance provisioning with data persistence

Inspired by @jeremy and @rachel I went ahead and wrote my first ever machine learning related blog post! It is on a set of scripts I wrote to make the bringing up and tear down of spot instances with persisting data as seamless as possible and you can find it here.

After you go through the guide, you will be able to run ./request-main-compute-instance.sh p2.xlarge 0.25 from your local box and subsequently you will be able to ssh into it by executing main-compute-instance-connect.

A persisted workspace volume will be automatically connected on provisioning and disconnected on shutdown. Shutdown happens also via issuing a single command from your terminal.

The IP address is also persisted so you can just go to a bookmark in your browser to access the jupyter notebook, the connection is over SSL, etc.

I based the scripts off the script provided for the first part of the MOOC. I wanted to make the experience as seamless as possible and the employed tagging mechanism offers a lot of flexibility so you can swap the persisted volume, AMI you use for the blue print of the instance, etc.

I have never done any bash programming in the past so the scripts are for sure rough around the edges, there is minimal error checking etc, but still I really like the convenience that they offer so thought I would share.

Maybe someone will find this useful :slight_smile: If you have any questions or problems or find any issues with the guide please let me know and I will try to help.

4 Likes