Salamander: AWS Spot Instances made easy


(Ashton Six) #1

tldr; $1.17 per hour for a V100 GPU & easier to use than Paperspace which charges double

Hi, I was inspired by your (the community’s) achievements in the DAWNBench competition, particularly the use of spot instances to lower costs. But even with the collection of scripts made available, running them still seemed really tough from a DevOps perspective. To make deep learning more accessible to aspiring researchers I quit my job and started building a product to make getting started easier and cheaper. Salamander adds just 26% to AWS spot prices (& reduces costs further with other optimisations). So with joy, I’d love to introduce Salamander to you all! :slight_smile:

Here’s a 1-minute demo:

And link to the website:

When designing a server, just select “fast.ai” under “Software” and all course materials + the library will automatically be installed into a new conda environment.

I don’t have enough money to give away free compute credits, but have done everything else within my power to lower costs. Your science and daily work are wonderful things, it’s through action that our love for humanity becomes apparent & I hope Salamander can help you all love humanity more.

& I’ll ofc be watching this thread to help, listen & talk. :slight_smile:


(Austin Shin) #2

Hi Ashton, do you still experience the slow initialization of the GPU on spot instances for first run? Do you have the option of specifying a custom AMI?

Thanks, great work.


(Ashton Six) #3

@penguinshin I’m not familiar with the slow initialization issue. Sometimes importing pytorch and calling .cuda() for the first time can be slow, perhaps related? Certainly less than 5 minutes at worst.

Regarding AMI’s, you cannot select a custom AMI (not built yet, but soon you’ll be able to initialise from custom docker images). Right now, you can select your desired software when designing a server; there’s 9 valid combinations and I created an AMI for each one:

screenshot of software selector


(Ashton Six) #4

@penguinshin

there are 3 reasons I chose not to support custom AMI’s:

  1. makes it harder/impossible to build advanced features like one-click launch for Jupyter Lab & notifications when the gpu stays inactive for over an hour
  2. Salamander only supports a subset of blockDeviceMappings, and therefore some AMIs will never work
  3. docker containers don’t have these limitations, but fulfil the same needs

(Austin Shin) #5

Makes sense. I guess the only limitation of docker is the installation time of the packages, but this is really only an issue for big packages like torch and cudnn, which you’ve already installed. Looks great!


(Ashton Six) #6

fyi @jeremy asked how Salamander compares to Crestle on Twitter & I’m now figuring everyone here might find that helpful too, so:

  • Both Crestle & Salamander make using Jupyter Lab really easy

Salamander Advantages

  • More instance types supported (12 vs 1) (max of 1,000 tflops vs 5.6)
  • Linux-first: you get normal AWS instances without weird abstractions
  • Crestle sessions are limited to 24 hours
  • Email notifications when instances are idle for over one hour
  • k80 hourly price is $0.36 vs $0.59 at Crestle
  • Storage price is $0.10 per gb/month vs $0.42 per gb/month at Crestle
  • I’m just getting started & have lots of cool features planned

Crestle Advantages

  • Has a web terminal
  • Infinitely scalable storage with zero configuration
  • New users get 1 hour of free GPU usage
  • Better support for python 2

I think Crestle is using EFS rather than volumes + snapshots for storage.


(Phani Srikanth) #7

This is awesome work @ashtonsix. Would you mind if I add Salamander to this list?


(Ashton Six) #8

@binga It’d be an honour to be included! Of course & thank you


(Jeremy Howard (Admin)) #9

That sounds like a big win - the latency of EFS was a real problem for Crestle.


(Ashton Six) #10

It’s worth noting snapshots also have latency problems, but not nearly as severe. When starting new servers your data gets gradually copied across in blocks. If you access a block that hasn’t been copied yet that’ll load across the network just like EFS. But subsequent requests for that data will be much faster. Don’t think it’s a problem for most workloads.

Salamander only keeps snapshots when instances are powered off to lower costs; also bc volumes can’t be attached to servers in different availability zones and we always want to pick the cheapest zone at startup.


#11

Hi there Ashton, thanks for this new website !
That seems awesome, unfortunately I might not understand how awesome it is, : how does adding " just 26% to AWS spot prices " can make it a better option than directly using AWS ?
Sorry, never used aws, did all the course on paperspace…


(Ashton Six) #12

@ouflepapi Great question! AWS’s on-demand price for a V100 is $3.06, and their spot price in the best availability zone is currently $0.93; Salamander adds 26% to that so $1.17. You can use spot instances directly but it’s much harder than the normal on-demand stuff; and if you make any mistakes there’s a risk of losing your work.

If you’d like to try using spot instances directly adapting the scripts fast.ai used to compete in the DAWNBench competition would be the best place to start (https://github.com/fastai/imagenet-fast/tree/master/aws), however I encountered the following issues when trying to do so myself:

  • You need to run this stuff manually from the command line, without a nice web interface
  • Mounting the primary volume after startup and using pivot_root has a small chance of failing when /init/sbin runs, which prevents the instance starting properly and can be hard to diagnose
  • Starting instances takes about 2 minutes longer because setting up the volume requires a reboot
  • Uses one particular availability zone, instead of picking the cheapest one
  • Keeps redundant volumes around when instances are turned off leading to increased storage costs (I think… but this might’ve been another solution I tried)
  • Some other stuff I can’t remember right now…

#13

That seems awesome indeed, thanks for the amazing work


(Cedric Chee) #14

Hi Ashton, I have give it a spin. I think, this is by far the most intuitive way of getting started (putting aside Google Colab and its restrictions, which are such a pain if you need to use it for more serious work). Great work!

Ditto. I used Terraform to automate and provision AWS EC2 Spot instance and followed Slav’s post here on how to swap the root volume. Yeah, the solution works but not perfect as what you mentioned there. But with Salamander, all these pain points are taken away. :sweat_smile:

Keep it up.


(Mario Lurig) #15

I recommend your list all the time. I would add the pricing estimate if possible. Currently, Salamander at the bottom of the list doesn’t seem attractive, but clearly, there are strong benefits for people to use it.


(Phani Srikanth) #16

Hey Mario,

Glad to know that you find it useful.
I was thinking about adding more metrics like pricing etc., but I refrained from it since I was not exactly sure what two metrics are the most important.

When I started out this project, I found no resource that aggregated all the providers out there and now we have one. Now that we have it, I am currently thinking about useful metrics like price / hour or any other metrics.

It might also be slightly tricky since the price that is quoted here should be for one particular flagship GPU instance so that price comparison makes sense (even though not all users are interested in this instance but can use this as a benchmark). Wanted to take a community input here. What metrics do you think will be most useful to users? Thanks for your time. :slight_smile:


(Ashton Six) #17

@binga Could we find time to get 93% accuracy on cifar10 or equivalent for each gpu, figure out which gpu is most cost-effective for each provider and use that value?

Only annoying thing that could throw of results is per-provider variation due to eg, available RAM & driver configuration.


(Dan Gerlanc) #18

Not sure if anyone has posted on this, but you can fully “initialize” your EBS volumes which have been restored from AMIs to avoid the slow initial performance (replace /dev/xvdf with your volume name):

sudo fio --filename=/dev/xvdf --rw=read --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize

This forces the instance to pull every block from S3 so it will require more time the larger your volume is.

See this AWS article for more info: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html


(Mario Lurig) #19

We should probably move any further replies to the github (as a courtesy). Doing so now.


(Phani Srikanth) #20

Time to 93% CIFAR could be useful but the variance of infrastructure between providers makes it slightly harder to draw an apples to apples comparison, as you rightly mentioned in your second point. Moving this conversation to the repo.