Xogroups: Running Standalone EC2 Instances on Spot

Alan Hand

February 18, 2019

Most Spot tooling assumes your workload is the kind of thing an Auto Scaling Group was built for: a horizontally scaled tier of identical, fungible instances behind a load balancer, where the answer to "instance i-abc123 just got reclaimed" is "ASG launched i-def456 in its place, traffic shifted, move on." That model works fine for stateless web tiers, batch workers, and the easy half of Kubernetes nodes.

It does not work for the other half of EC2 capacity in a real production account.

Replicated Kafka clusters. Self-managed Cassandra, Elasticsearch, or MongoDB clusters. Clustered application servers that don't auto-scale and run a fixed number of instances. HAProxy and Nginx pairs that share traffic via DNS or VRRP. Legacy services that someone built years ago and nobody is going to refactor into an ASG anytime soon. These workloads live outside an ASG, often by necessity. They have a static private IP that other systems depend on. They have an Elastic IP attached to their network interface. They have an EBS volume with state on it. When you "replace" one of these, you can't just terminate it and let an ASG launch a fresh one somewhere else: the new instance would have a different IP, a different ENI, an empty disk, and a five-page list of things that broke downstream.

Xogroups are how Xosphere extends Spot replacement to this class of workload.

What is an Xogroup

An Xogroup is a set of standalone EC2 instances that you've designated as a unit for Xosphere to manage. There is no new AWS resource and no agent on the instance. Xosphere orchestrates Spot replacement across the members of the group, swapping individual instances onto cheaper capacity while keeping the workload available.

When the orchestrator decides to replace an instance in an Xogroup, three things happen that don't happen with an ASG:

1. Group membership is declared with a tag, not a resource. ASG membership is structural: an instance is in an ASG because the ASG resource created it and tracks it in its desired-count math. Xogroup membership is declarative: an instance is in an Xogroup because it carries the tag xosphere.io/instance-orchestrator/xogroup-name=<your-group-name>. Any two instances with the same tag value are in the same group. Any standalone EC2 instance can join, leave, or move between Xogroups by changing a tag value. There is no ASG to refactor your IaC around, no terraform diff that destroys and recreates anything, no migration plan.

2. Replacement is a rolling restart, not a scale-out then scale-in. ASGs replace by launching a new instance somewhere else, waiting for it to come up healthy, then terminating the old one. The application sees a brand new host with a new IP, a new ENI, and an empty local disk; anything downstream that cared about the old instance's identity has to re-converge. Xogroups instead gracefully remove an instance from service, then bring it back as if it had rebooted. To the application, its host went down briefly and came back. No new host appeared from downstream's perspective.

3. The replacement preserves the instance's identity. The "rebooted" instance comes back with the same private IP address. The same Elastic IP. The same EBS volumes attached at the same mount points. The same ENI attachments with the same security group rules. The same user data, key pair, kernel ID, EBS optimization flag, monitoring state, shutdown behavior, and API termination/stop protection settings. From the application's perspective and from anything downstream that cared about the instance's network identity or local disk, nothing changed except the underlying physical host.

What identity preservation actually means

The third property is the one that does the most work, so it's worth being concrete. When the orchestrator replaces an instance in an Xogroup, it goes through this sequence:

  1. Snapshot the source instance's identity. Capture the running configuration: instance type, AMI, EBS volume IDs and their DeleteOnTermination flags, network interfaces with their private IP allocations, Elastic IP associations, user data, key pair, kernel and RAM disk IDs, EBS optimization flag, monitoring state, shutdown behavior, API termination and stop protection settings, and tags.

  2. Gracefully remove the instance from service. Deregister from load balancer target groups it's attached to, drain in-flight connections, and give the application its expected shutdown signal so it can flush state cleanly. Whatever drain hooks the application has are honored.

  3. Disable DeleteOnTermination on the data volumes. Before terminating the old instance, the orchestrator flips DeleteOnTermination=false on the EBS volumes that should survive. The volumes detach when the instance terminates rather than getting deleted with it.

  4. Terminate the old instance. With the stop and termination signal the operating system expects. This is the "reboot" from the application's perspective.

  5. Launch the replacement. A new EC2 instance with a different instance type (a Spot price-optimal one, by default) but the same AMI, same user data, same key pair, same kernel and RAM disk, same EBS optimization, same monitoring, same placement, same shutdown behavior, same API stop and terminate flags. The launch reuses the network interfaces from the old instance, so the private IP is preserved. The volumes that survived step 3 get attached at the same device names.

  6. Re-associate the Elastic IPs. Any EIPs that were on the old instance get re-associated to the new one, mapped to the same private IPs via the same ENIs.

  7. Restore the DeleteOnTermination settings. Volumes that were marked for delete-on-terminate originally get that flag put back, so future terminations behave the way the operator originally intended.

  8. Re-register and verify. The orchestrator re-attaches the instance to any target groups it was registered with and confirms the replacement is up, network is attached, volumes are mounted. If verification fails, the run is held and the operator is alerted rather than blindly proceeding to the next instance.

A workload running on the box does not see a new network identity or a new disk. It sees its host process restart. Anything that depended on the instance's private IP, EIP, or attached storage keeps working.

Where Xogroups fit

The question to ask is not "is my workload stateful?" but "can I take one instance out of service for a few minutes without breaking the service?"

If yes, the workload is an Xogroup candidate. Examples:

  • A Kafka or RabbitMQ cluster with replication factor at least 2 and rack-aware partitioning. Stop one broker, let the cluster rebalance, bring it back in. Standard cluster admin operation.
  • A self-managed Cassandra, Elasticsearch, or MongoDB cluster sized so that a single node loss is recoverable.
  • A pair of HAProxy or Nginx instances sharing traffic via DNS health checks, where one going out of service for a few minutes means traffic shifts to the other.
  • A clustered application tier with a fixed number of instances that doesn't auto-scale. Common for legacy systems that were never written with horizontal scaling in mind, or for workloads where capacity is sized once and left alone.
  • Legacy services that need to keep running but won't be refactored into an ASG. The instance was hand-rolled, has accumulated state, and the team's effort is better spent elsewhere. Xogroups let those workloads benefit from Spot without a rewrite.

If no, the workload is not a good Xogroup candidate, with one useful exception: development and test environments where a few minutes of downtime per replacement is acceptable. A single-replica internal tool that nobody uses on weekends is fine to put on Spot via an Xogroup. The savings still apply, the downtime risk is fully contained, and most teams find that the "outage" goes unnoticed.

Workloads that genuinely don't belong in an Xogroup:

  • A production single-replica database with no read replica or warm standby. There is nothing to fail over to during the replacement. Even a 60-second outage is an outage.
  • A workload with a multi-minute warm-up that's load-bearing for hard SLAs. The replacement comes up cold; if cold start blocks traffic for five minutes, that's five minutes of impact.

For those, keep them On-Demand or back them with Reserved Instances.

How you'd actually set one up

The setup is deliberately undramatic. Tag the instances:

Tags:
  - Key: xosphere.io/instance-orchestrator/xogroup-name
    Value: kafka-prod
  - Key: xosphere.io/instance-orchestrator/xogroup-enabled
    Value: true

That's the minimum. Optional tags let you constrain which instance types the orchestrator may choose as replacement candidates, for example restricting a Kafka broker to instance types with the same vCPU and memory shape, or excluding types with insufficient network bandwidth. Configuration lives on the instance itself, alongside everything else that already describes it.

There is no new control plane, no agent inside the OS, no inbound network change. The instance keeps being a regular EC2 instance, addressable by the same private IP, owning the same EBS volumes, advertising the same EIP. Xosphere acts on it through the AWS API the way any other automation would.

Tradeoffs worth naming

A few realities of running stateful workloads on Spot via Xogroups:

  • Cluster-level safety is the application's responsibility. The orchestrator replaces one instance at a time, but it does not understand whether your Kafka cluster currently has all partitions replicated, or whether your Cassandra cluster is mid-repair. Use the standard health checks and operational guardrails your data system already provides; Xogroups don't override them.
  • EBS snapshot strategy still matters. Volumes persist across the replacement, which means they also persist their state, including any state you'd rather have reset. Treat snapshot cadence the same way you would on On-Demand. The replacement does not back up your data for you.
  • The replacement is still an event. It's brief and the application sees it as a reboot rather than a fresh host, but it is a reboot. Workloads that can't tolerate that within their normal operational envelope shouldn't be on Spot regardless of orchestration model.

Spot for the workloads that couldn't run on Spot before

For years, the conventional advice has been to put stateless tiers on Spot and leave everything else on On-Demand or Reserved Instances. That advice is correct when the only orchestration available assumes interchangeable instances behind a load balancer. It's also where most production EC2 spend sits stuck.

Xogroups change the math. Replicated brokers, clustered databases, fixed-size application tiers, the legacy systems that have been quietly running on On-Demand because no one had a safe way to move them: these can run on Spot now. The savings compound across the parts of the bill that were previously out of reach, which for most accounts is a meaningful share of total EC2 cost.