The only problem with spot instances is that it might be terminated if market price is higher than your bid price. But if you setup cron task to sync all your stuff to somewhere like s3 - you can save a couple of $ (for non GPU instances typically 50%).
I guess the most common mode of operation would be tunneling via Jupyter…what happens when there is an interruption to the Spot Instance though?
When Amazon EC2 interrupts a Spot Instance, it provides a Spot Instance interruption notice, which gives the instance a two-minute warning before Amazon EC2 stops or terminates it. You can’t enable termination protection for Spot Instances.
When a spot instance is shut down for any reason, you lose all data on it. Avoiding this requires some tricks. There’s a lot of info about this on #part1 so I’d suggest heading to those threads if you’re interesting in this topic. FYI crestle.com uses spot instances behind the scenes, but manages all this for you.
As @jeremy wrote - you lose all your not synced stuff. If you sync it once per minute, you might lose only some not finished calculations and I assume we not gonna train models from scratch so it is safe. Yes you need to install all libraries and everything and sync back your data/files, but with scrits prepared it is a matter of minutes.
I also worked out my own solution which I describe in a medium post Not sure how it compares to the solution by @slavivanov but it allows me to spin up a spot instance of my choice issuing a single command from terminal on my local machine. The way this works is that you have a volume that is getting attached while you provision the machine and that gets detached at tear down so you don’t lose your work.
All we need to get this to work for part 1 v2 is a setup script to replace the one provided for part 1 v1. @jeremy mentioned in one of the other threads that maybe he will be releasing one - which would be great as I would prefer to use something solid - and if not I will modify the script from part 1 v1. Have something rudimentary here but IIRC there were some issues with it.