Pre-emption Explained
Pre-emption only applies to Spot instances, On-Demand instances never face Pre-emption risk. See Spot vs On-Demand
What is Pre-emption?
Rackspace Spot matches available cloud capacity with bids, by using an Open Market Auction. During the auction, existing bids for capacity at lower prices may end up "losing out" to other higher bids for capacity. In order to fulfill the higher value bid, Spot will reclaim that existing capacity to fulfill the market auction. This reclamation of capacity results in cloud servers being "pre-empted", with those lower bid servers being terminated.
Understanding when and why pre-emption happens, and how to estimate the risk, is crucial for effectively using Spot Instances.
Why Does Pre-emption Happen?
The trigger for pre-emption is if Market Price Exceeds Your Bid:
- The price for Spot Instances fluctuates based on real-time supply and demand in the open market auction. You set a maximum bid price, the most you're willing to pay per hour for an instance. If the market price rises above your maximum bid price, your instance is flagged for pre-emption.
Your bid sets your price ceiling. You only pay the current market price while your instance runs, as long as that market price is at or below your maximum bid. Pre-emption occurs when the market price surpasses your bid.
Avoiding Node Pre-emption
The simplest and easiest way to avoid node pre-emption is to bid higher for your requested cloud servers. Note that market price is subject to fluctuations, and there may be periods when the price is low but there could also be short windows of time when the price is higher than you may expect.
Since Rackspace Spot bids are always billed at the prevailing market price, bidding higher doesn't necessarily mean that you will incur a high cost over a period of time; but rather it serves to avoid your bids being pre-empted by a short term price fluctuation caused by external events - e.g. prime time, week of day, and other seasonal factors.
Fairness in Pre-emption
The Open Market Auction allocates capacity from higher priced bids to lower priced bids. Therefore, the criteria being used for pre-emption is nothing but the maximum bid price.
In addition, when there is a "tie" between an existing bid that is using nodes, vs a newly incoming bid for new cloud capacity, Spot will avoid interrupting the existing user of capacity. The onus is on new requests for capacity to bid higher than the current market price to be allocated capacity.
Pre-emption Process
In the event your bids are pre-empted, Rackspace Spot tries to do so in an orderly manner that is minimally disruptive to your application, by:
- First, issuing a notice of "Upcoming Pre-emption" to a user provided webhook. The nodes in question are now cordoned to prevent scheduling of new workloads on these nodes
- Then, 1 minute later: Affected nodes are drained, to allow any running pods to terminate in an orderly manner
- Then, 5 minutes later: A second notice, "Pre-emption", is sent out when the node is ejected from the cloudspace
Getting Notified On Pre-emption
You can choose to be alerted when nodes are pre-empted. To do so, enable the Cloudspace option that says "Alert on Pre-emption", and provide the webhook where you would like to receive the notification. As of today, we support Slack webhooks.
To create such a webhook, please see:
Estimating Risk: Capacity Availability Insights
While pre-emption is always a possibility with Spot Instances, you can estimate the risk based on current market conditions using the Capacity Availability insights provided in the Rackspace Spot dashboard.
This data shows the relationship between bid prices and the percentage of available capacity for a specific instance type in a region at that point in time.
Using the Capacity Availability Metric
For each server class click "What should I bid?" to view price point metrics relative to available capacity:
Bids by Capacity Availability metric for a certain server class
Understanding Price Percentiles
The graph and associated data often show key price percentiles (e.g., 20th, 50th, 80th) relative to capacity. These help gauge how competitive your bid needs to be:
| Price Percentile (Approx.) | Interpretation (Relative to Current Market Depth) | Estimated Pre-emption Risk |
|---|---|---|
| Market Price | Price where ~0% capacity is available | Imminent/Very High |
| 20% Capacity Available | Price is high enough to access only ~20% of current available capacity | High |
| 50% Capacity Available | Price is high enough to access ~50% of current available capacity | Moderate |
| 80% Capacity Available | Price is high enough to access ~80% of current available capacity | Low |
Recommendation:
To minimize the initial risk of pre-emption when launching nodes, consider bidding at or slightly above the price associated with 80% capacity availability. Remember, the market is dynamic, so this risk level can change over time.
Using Price Insights Effectively
This data helps you:
- Assess Market Competitiveness: Compare different server classes "Bid by Price History" metric. Flatter price curves often indicate less competition than steep curves.
- Position Your Bid: Understand the potential risk associated with different bid prices based on current market depth.
- Monitor Market Changes: Periodically check if market shifts have changed the risk profile for your existing bids. A bid that was previously low-risk (at 80%) might become higher-risk if overall demand increases.
Strategies for Managing Pre-emption Impact
While you can estimate and influence risk through bidding, the most robust way to handle pre-emption when it occurs is through application design and cluster configuration:
- Fault Tolerance: Design applications to withstand unexpected node failures (e.g., run multiple replicas, use leader election, checkpoint progress).
- Statelessness: Favor stateless application components where possible, as they are easier to reschedule onto new nodes.
- Hybrid Clusters: Run critical, stateful, or interruption-sensitive workloads on On-Demand Instances within the same Cloudspace.
See more details for Bidding Best Practices
Programmatic Access to Capacity Insights
For advanced users and automation, the price and capacity insights shown in the UI are also available programmatically via a JSON endpoint:
Endpoint: https://ngpc-prod-public-data.s3.us-east-2.amazonaws.com/percentiles.json
This allows you to build tools that monitor market conditions and potentially adjust bids automatically.
Example Data Snippet:
{ "regions": {}}