8.9.10.1.2 - Spot Instance Interruptions: Losing Your Server Mid-Process to Save Costs (Difficulty: Hero | Path: Lab)

Imagine renting a Ferrari for $10 an hour instead of $100, with one specific catch: the rental agency can call you at any moment and demand the car back within 120 seconds, regardless of whether you are parked in a driveway or speeding down the highway at 100 mph. This is the fundamental trade-off of AWS Spot Instances (and their equivalents on Google Cloud and Azure). Cloud providers have massive amounts of idle server capacity—massive GPU farms sitting unused—so they auction this excess capacity off at steep discounts, often between 60% and 90% off the On-Demand price. For an AI engineer training large models, this discount can mean the difference between a $500 training run and a $5,000 one.

However, this financial arbitrage comes with a severe operational volatility known as Spot Instance Interruption. Because you are essentially using "standby" capacity, you do not own the slot. If a full-paying customer arrives and needs that specific GPU, or if the cloud provider decides to reshuffle capacity for maintenance, you are evicted. The system sends a termination signal (usually giving you a scant two-minute warning), and then the server vanishes. It doesn't just pause; in many configurations, the instance is terminated, and the local ephemeral storage is wiped clean.

For a web server, this is a minor annoyance; the load balancer simply redirects traffic to another node. But for a Deep Learning workload—where a model might be 40 hours into a 50-hour training epoch—an interruption is catastrophic if not handled correctly. If your training process relies solely on the server's RAM or local NVMe drive to hold the model weights, an interruption at Hour 39 means you have lost 39 hours of compute time and the money you paid for it. It is the digital equivalent of a power outage unsaving your game right before the final boss.