When 99.99% Matters: High Availability Patterns for Engineers

Core high availability patterns: active-active, active-passive, replication and failover

All rock-solid architectures are built on a handful of well-known patterns. The tradeoffs between them are: failover time, consistency, and operational overhead.

Active-active versus active-passive is the first fork in the road. Active-active runs multiple nodes simultaneously, all of them accepting writes, all of them serving traffic. Active-passive keeps one node live and one or more standbys idle, ready to take over. Redis’s engineering team notes that active-passive “will usually incur a short failover delay (on the order of a few seconds to a few minutes) while active-active will be able to provide almost immediate failover with no promotion step involved”. The catch is write-conflict resolution: when two nodes accept writes to the same record simultaneously, you need conflict resolution logic (usually last-write-wins, vector clocks or CRDTs). That’s the complexity reason why active-active gets reserved for the workloads that truly cannot tolerate a gap.

Replication is the basis of both patterns, and your choice between synchronous and asynchronous is the factor that determines your recovery point objective (RPO) directly. Synchronous replication won't acknowledge that a write has succeeded until it is confirmed on every replica, so your RPO is virtually zero, but the latency cost of every write is the round-trip time across the distance to your replicas. Asynchronous replication will locally acknowledge the write and replicate it later, so latency is lower but a crash can lose everything that had not yet been replicated. Academic pattern-language work on high availability records lazy replication as a conscious trade of consistency for throughput, not a failure, once you've sized the window of acceptable data loss beforehand.

Load balancing makes the pattern work. Health-check-driven routing determines which nodes actually receive the traffic, and the difference between liveness and readiness probes matters more than most teams realise. A liveness probe asks "is this process alive?" and restarts the process on failure. A readiness probe asks "can this node currently serve requests correctly?" and should just pull the node from rotation without restarting it, since restarting a node that's merely overloaded or waiting on a slow dependency only makes things worse. Distributed systems design guidance calls this the two-level health check, and omitting the distinction is a common source of what practitioners call grey failures: nodes that are technically up but functionally broken.

Clustering topology is the final decision point:

• Shared-storage clustering allows the failover nodes to mount the same disk that the primary was using, so there is no need to resync the data on promotion. However, this makes the shared storage layer into another dependency that you must also ensure is highly available.
• Shared-nothing clustering provides each node with its own storage, using replication to synchronize them. This trades resync time for the removal of a shared bottleneck.

N+1 redundancy involves one spare unit over the minimum required, a pattern with roots in hardware engineering. This pattern can be effective when failure domains are independent and hardware is heterogeneous enough that "N+1 identical spares" is not true in all cases.

Aligning pattern to availability tier keeps decision rooted to the ground. Three-nines (99.9%) targets are typically served by active-passive with automated failover behind a single zone. Four-nines (99.99%) generally requires multi-zone deployment with health-check driven load balancing. Five-nines (99.999%) is where active-active multi-region pays its complexity tax, because no single-zone or single-region pattern can plausibly reach that target in a scenario where an entire region goes dark.

Resilience building blocks that keep dependencies from taking you down

Primary patterns cover node and zone failure. They do not cover the more common failure mode: a dependency several layers down becoming slow or unresponsive, and that slowness cascading up until your entire request path is blocked waiting on it.

Four supporting patterns close that gap, and Microsoft’s Azure Well-Architected reliability guidance names them as standard building blocks:

• A circuit breaker prevents a failing dependency from receiving requests after a threshold of errors by returning a fast failure or fallback instead of queuing up requests and exhausting threads.
• Cache-aside provides stale-but-available data from cache while the source system is down. It's a tradeoff of freshness for availability during an incident.
• Queue-based load levelling flattens out traffic bursts by queuing work rather than losing requests when downstream capacity is exceeded.
• Graceful degradation intentionally turns off non-critical functions (recommendations, personalisation, analytics) to ensure core features continue to work during a partial outage.

Names are important, but the details of the implementation are equally important. A circuit breaker needs an appropriate fallback (ideally not an error) to return; a cached response or default response is often preferable to an empty error page. Cache-aside must have an explicit staleness policy: how stale is too stale for this data type, and does the user need to be aware that it is stale. Queue sizing must have headroom for the worst plausible burst, not the average burst, or the queue becomes the bottleneck during the very event it was intended to mitigate.

Measuring availability: nines, SLOs, RTO and RPO

This “nines” convention carries over directly to downtime budgets, and the numbers are lower than most folks realize. The Wikipedia high availability reference article puts 99.9% availability at about 9 hours of downtime a year, 99.99% at about 53 minutes a year, and 99.999% at about 5.3 minutes a year. That last figure is the one that reframes the conversation: five-nines gives you about as much time to notice an alert, let alone manually intervene, which is why five-nines targets demand automated failover regardless of how mature your on-call process may be.

Deployment scope should flow directly from your objective. Google Cloud's infrastructure reliability documentation makes the same point, associating 99.9% with single-zone deployments, 99.99% with multi-zone, and 99.999% with multi-region. The AWS Well-Architected reliability pillar echoes this framing, emphasizing that "multi-AZ is not the same thing as multi-region, which provides additional isolation to protect against the regional risks that multi-AZ cannot mitigate."

RTO (how long recovery takes) and RPO (how much data you can afford to lose) should dictate choice of replication and failover pattern, rather than the converse. A five-minute RTO will eliminate any pattern involving manual DNS changes and manual promotion. A near-zero RPO eliminates asynchronous replication entirely. When RTO and RPO are settled upon as precise numbers rather than as ideals, the choice of pattern often makes itself.

SLOs and error budgets provide you with a spending mechanism for all of the above. If your SLO allows 53 minutes of downtime a year and you've used 40 minutes by September, that's your signal to stop shipping risky changes and invest your remaining budget in stability, instead of treating every outage as a discrete event unrelated to the annual goal.

A decision framework for choosing patterns without over-engineering

My experience is that most examples of over-engineering didn’t come about maliciously. They just happened because somebody looked at the business requirements and then reached for the most resilient pattern on the shelf before writing down the actual business impact of downtime. Work through each of these in order, and fight the urge to skip ahead to step 4:

1. Quantify business impact first. How much does an hour of downtime cost, in lost revenue, SLA penalties, or reputation damage? A B2B invoicing system and a real-time trading platform have very different answers, and that number should be on paper before any architecture discussion starts.
2. Define RTO and RPO based on the business impact, not technical preference. If an hour of downtime is a rounding error, 30 minutes RTO is perfectly fine. If an hour of downtime costs a contract, RTO needs to be in seconds.
3. Match pattern and deployment scope. Single-zone active/passive to hit three-nines targets. Multi-zone with health-check-driven load balancing for four-nines. Multi-region active-active only when the RTO really requires sub-minute recovery from the loss of an entire region.
4. Estimate the cost delta honestly. Multi-region active-active roughly doubles infrastructure spend and adds ongoing engineering cost for conflict resolution and testing. Compare that number against the downtime cost from step 1, not against a vague sense that “more redundant is always better.”
5. Gauge your team's operational maturity. Do you have automated failover in place, or does someone need to be paged at 3am to run a manual runbook? Multi-region active-active without mature automation and well-rehearsed runbooks tends to generate more outages, not fewer, because the extra complexity itself is a source of failure.
6. Favor the simpler, proven pattern. A lot of AWS’s documentation around resiliency trade-offs is advocating for static stability, systems that don’t need the control plane to be available to them in order to recover, simply because the kinds of complex, active-active, RIAK-STUN-studded recovery paths you’d need to make such a system work tends to die during the exact large-scale outage it was architected to withstand. If your number from step 1 doesn’t require it, active-passive with good automation beats active-active with iffy automation nearly every time.

The value of the framework is that it requires steps 1 and 2 to be completed before step 3. Skipping the impact analysis, you end up building five-nines infrastructure for a system that only needed three. That is, costly redundancy that does not reduce the real business risk.

Implementation checklist: health checks, automated failover and testing

A pattern on a whiteboard means nothing unless it passes a real failure. Build the implementation around these four steps:

1. Design liveness and readiness checks separately. Liveness failures result in a restart; readiness failures pull the node out of the load balancer pool but don't restart anything. Confusing the two is one of the most common causes of unnecessary outages, because a node that's merely busy gets killed instead of temporarily bypassed.
2. Fully automate the detect, promote, verify loop. As in step 1, detection should automatically cause promotion within seconds. The promotion should in turn cause an automated verification step (health check against the newly-promoted node) before all traffic is moved. Route 53's failover routing policies, documented at http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover.html, is one (DNS level) way to implement active-active and active-passive configurations at the traffic-routing layer. Weighted routing is a nice halfway measure, enabling a more gradual rather than all-or-nothing cutover.
3. Build a testing matrix that covers realistic failure combinations, not just the easy single-node case: single node failure, full AZ failure, full region failure, degraded (not dead) dependency, and split-brain simulation where a network partition briefly convinces two nodes they're each the primary.
4. Put chaos engineering rehearsals on a calendar, not "when we get to it." Killing a node on a controlled game day reveals things a design review never will, including whether your alerting actually fires and whether the runbook you wrote six months ago still maps to the current system.

Trade-offs and anti-patterns that quietly undermine availability

Redundant compute nodes are worthless if all traffic still flows through a single unprotected control plane, DNS provider, or load balancer. That’s the invisible single point of failure present in nearly every post-incident report: the redundancy was in place, but it was downstream of a dependency that nobody replicated.

Split-brain warrants special attention. Network partitioning the cluster can leave both halves believing they're primary unless quorum logic with a witness node intervenes, and a misconfigured quorum can make a trivial partition become a data-corruption event.

Complexity is not free. It's not just a matter of design tradeoffs. Each additional pattern that you bolt on is another component that can fail and another thing you have to test regularly. Teams that add active-active replication without the matching testing discipline find themselves less reliable than the simpler system they replaced. DNS and network-layer availability, DDoS hardening and the like gets neglected constantly because it falls outside the normal application team's scope, right up until it's the cause of a perfectly-healthy backend becoming unreachable.

How PODTECH builds high availability into mission-critical systems

PODTECH supports its enterprise software delivery with a 99.9% uptime SLA and follow-the-sun support, because critical infrastructure clients can’t absorb a support gap during a failover event. Engagement models (dedicated teams or staff augmentation) embed HA thinking from the architecture phase, not bolted on after launch. Design reviews on datacentre and BMS/PMS integration projects work through RTO/RPO mapping, documented runbooks and a record of failover test history before sign-off, drawing on delivery experience across 250-plus projects in mission-critical sectors.

Choosing your pattern: the short version

Align your availability target with the simplest pattern that will achieve it: active-passive for three-nines, multi-zone with health checks for four-nines, active-active multi-region only when five-nines is really, really needed. Three things to do this week: define explicit SLOs including an error budget, choose your deployment scope, and conduct one failover rehearsal to discover what your runbook actually fails to cover.

Why most teams pick the wrong pattern for the wrong reason

The standard narrative on high availability has active-active as the "pie in the sky", something that only real "big boys" should work toward. Wrong. The data on failover requirements is pretty clear: active-passive with automated failover meets the vast majority of real business RTOs, and the seconds-to-minutes difference that active-passive entails is almost invariably immaterial compared to the write-conflict overhead that active-active extracts.

The underrated selection criterion is operational maturity. Architects scale patterns based on availability targets and cost, and rarely against whether the on-call team has ever rehearsed the failover they’re designing. A five-nines architecture operated by a team that has never failed a region is not five-nines. It is three-nines with an expensive story attached.

Do the boring work first: have real numbers for RTO and RPO, a set SLO with error budget, have run the rehearsal before buying the complexity. Pattern selection should be a piece of cake at that point.

— Harry

Sources

For more comprehensive treatment of these concepts than is provided in this article, see AWS's Well-Architected reliability pillar, Google Cloud's infrastructure reliability guide for cloud-native design patterns, Microsoft's Azure Well-Architected reliability patterns for building block patterns for resilience, and the proceedings of the PLoP pattern language workshops for the academic underpinnings of the replication trade-offs.

• Active‑Active vs Active‑Passive (Redis blog)
• Design patterns — reliability (Microsoft Azure Well‑Architected)

Recommended

• Critical Infrastructure Datacenter Solutions
• Custom Software Development in Datacenters: The Single Pane of Glass Imperative
• Datacenter Mobilisation: Why Monitoring Teams Must Be Involved from Day One
• Datacenter Services - PODTECH