Yes, you can run stateful workloads on Kubernetes in production. In fact, most organisations do. The hard part is not getting a database pod to run. It is about maintaining identity, storage, placement and day-two operations like backup, failover and patching when things go wrong. Nodes fail. Zones go dark. This guide walks through what Kubernetes gives you out of the box, where it falls short, and how we tackle the rest.
Build software that supports vital infrastructure
PODTECH creates custom software for infrastructure that cannot afford to fail, whether that is scalable platforms, automation technology, or telemetry for buildings. Learn More
TL;DR
- Use a StatefulSet rather than a Deployment for any workload that requires stable networking identity or its own persistent volume.
- Storage is where the real design decision happens. Topology-aware provisioning and spread constraints ensure replicas and volumes land in the correct failure domains.
- StatefulSets themselves will not manage your database. Failover, schema changes and restores require an operator or solid runbooks.
- Watch the right metrics. Readiness, disk usage, restart count and replication lag are better indicators than total pod count.
- Decide workload by workload. Some data should live on the cluster, and some should not.
Table of Contents
- What makes a workload stateful
- Stateless versus stateful: what changes
- What a StatefulSet actually guarantees
- Storage is the design decision
- Placement: keep replicas apart, keep volumes reachable
- Day-two operations: where operators come in
- Should this workload be on the cluster at all?
- Stateful workloads at the edge and in the building
- Monitoring stateful workloads
- Our perspective
- Where PODTECH can help
- A pre-production checklist
- FAQ
- Sources
What makes a workload stateful
You can kill a stateless service and bring up an identical replacement without any real ramifications. You cannot do that with a stateful service. Stateless services hold no data that must outlive a restart. They also do not rely on a stable identity: a hostname others use to look them up, a place in a cluster, or a status as primary or replica.
Databases are the canonical example. Message brokers, search indexes, distributed caches with persistence, and coordination services exhibit the same behaviour. Many industrial databases do as well: time-series stores for building telemetry, event logs from safety systems, and historian databases driving analytics.
Demand keeps growing. In the 2020 CNCF survey, 55% of respondents said their organisations were running stateful applications in production, and another sizable fraction were considering or planning to. Databases came in first place. The question is no longer if, but how well.
Stateless versus stateful: what changes
| Concern | Stateless (Deployment) | Stateful (StatefulSet) |
|---|---|---|
| Pod identity | Interchangeable, random names | Stable, ordinal names such as db-0, db-1 |
| Storage | Ephemeral or shared | One persistent volume per replica |
| Scaling and updates | Parallel, any order | Ordered, one at a time by default |
| Failure response | Replace and forget | Replace and reattach the same data |
| Recovery logic | Rarely needed | Backup, restore, failover, resync |
| Placement | Anywhere with capacity | Constrained by where the volume lives |
Each line on the right represents an operational responsibility. That is the price of running stateful systems, and why the platform matters less than the discipline imposed around it.
What a StatefulSet actually guarantees
The Kubernetes documentation describes a StatefulSet as a controller that runs a specified number of Pods and manages the ordering and uniqueness of those Pods.
In practice, that translates to four guarantees:
- Stable, unique network identifiers. Pods are assigned predictable names, and when paired with a headless Service, predictable DNS entries as well. Members of a cluster can locate each other across restarts.
- Volumes persist across rescheduling. A replacement pod mounts the same volume as the pod being replaced.
- Ordered, graceful deployment and scaling. Replica 0 deploys before replica 1, and so on. Scale down happens in the opposite order.
- Ordered, automated rolling updates. Rolling updates take place one pod at a time so that a replicated database application can maintain quorum.
These guarantees revolve around identity and persistence. They say nothing about how your application interprets them. A StatefulSet will happily restart a corrupt primary and reattach the corrupted disk.
Volume claim templates
The volumeClaimTemplates field is what makes per-replica storage work. Every replica gets its own PersistentVolumeClaim created from that template. The claim is named using the template name, the StatefulSet name, and the ordinal. That means postgres-1 reattaches to its own claim if it gets rescheduled, not its neighbour’s.
Those claims are deleted separately by default when pods die. Scale down a StatefulSet and the data remains. This is the safe default for critical data, but orphaned volumes do not go away and continue to cost money. Kubernetes introduced a PVC retention policy that lets you decide separately, for StatefulSet deletion and for scale down, whether to keep or delete claims. Deletion should only be used when the data can be recreated from another source, such as a cache or a replica set that resyncs.
Storage is the design decision
The majority of production outages affecting stateful workloads are storage-related outages. Practising a few simple disciplines prevents a surprising number of them.
Choose your storage class intentionally. Latency, IOPS, snapshotting and volume expansion vary greatly by class. You would not store a write-intensive database and a log archive in the same bucket by accident.
Know what attach and mount failures look like. Pods can get stuck in ContainerCreating with attach or mount errors because the volume they want is still attached to a node that is no longer reachable. Our provider-specific troubleshooting notes on persistent volume issues include examples of what those events look like and explain why it is almost always a storage-layer issue rather than something wrong in your app. Familiarise yourself with them early.
Restore, not backup. Testing restores verifies backup. If you have never restored a snapshot, it is only a hope. Restore into a scratch namespace and measure the result against your recovery time objectives.
Expansion plan. Ensure your storage class supports volume expansion before your database fills its disk at 3 a.m.
Placement: keep replicas apart, keep volumes reachable
Volumes generally exist in a single zone. If the scheduler places a pod in a different zone from its volume, the pod will not start. If it places all replicas in the same zone, a zonal outage can take down your entire database.
Two features address this:
- Topology-aware volume provisioning. Kubernetes waits until the pod is scheduled before binding the volume. This allows the platform to provision the volume in a zone where the pod is actually allowed to run. The Kubernetes project blog post on this feature describes it as making smart decisions when dynamically provisioning volumes in multi-zone clusters.
- Pod topology spread constraints. Schedule pods evenly across your infrastructure so replicas do not collapse into the same failure domain.
Why not do both? Set the storage class to wait for the first consumer. Add a spread constraint across zones with a hard limit on skew. Add anti-affinity or node spread so two replicas are never colocated on the same host. Then try draining a node and cordoning a zone in a staging cluster and see what happens.
One limitation remains: volumes attached to a pod in one zone cannot follow that pod to another zone. Cross-zone high availability must be provided by replication at the application level, where each replica manages its own volume.
Day-two operations: where operators come in
StatefulSets provide identity and storage. They do not manage the operational responsibilities required to keep a database healthy: automatic failover when the primary becomes unhealthy, schema migrations, consistent backups, certificate rotation, or re-indexing.
The Data on Kubernetes 2022 report quantifies that gap. It identified automating provisioning and day-two lifecycle tasks as the top pain point organisations experience when running data on Kubernetes, and operators as the solution. An operator is a controller that codifies a human operator’s knowledge into the Kubernetes API. You declare a database cluster, and the operator provisions the StatefulSets, services and backups, then responds to failures.
When evaluating an operator, ask:
- Is automated failover supported, and has it been tested against node loss and network partitions?
- Does it support point-in-time recovery and restore into a new cluster?
- How does it upgrade the database engine and itself?
- Does it expose metrics you can alert on?
- Is it actively maintained, and does its licence fit your use?
There is an economic argument here too. Operators can enable teams to run database-as-a-service internally, which some argue leads to cost savings, portability and consistency rather than relying solely on managed services from individual cloud providers. That is only one perspective, of course, and the math depends on your team’s expertise, so run the numbers against your own costs.
Should this workload be on the cluster at all?
Not all data deserves a home on Kubernetes. The right balance comes from evaluating each workload individually.
| Question | Leans towards the cluster | Leans away |
|---|---|---|
| Does the team have operator and storage expertise? | Yes | No, consider a managed service |
| Is portability across sites or clouds required? | Yes | No |
| Are latency and data locality tight? | Yes, with local or fast storage | Dependent on a remote service |
| Is the engine well supported by a mature operator? | Yes | Niche or unmaintained |
| Are recovery objectives strict and tested? | Tested | Untested |
This exercise is best run workload by workload rather than company by company. A useful model appears in a post on deciding between on-premise and cloud workload by workload. Even if the article itself is in French, the trade-off framing is directly usable.
Stateful workloads at the edge and in the building
Many enterprises need data far from headquarters to drive their businesses. Power meters, cooling loops, leak sensors and access systems all generate critical data at edge locations. Local clusters at enterprise edge sites can cache and retain telemetry so that a severed WAN link does not delete it. These requirements make the edge database a stateful workload with unique constraints: tiny footprints, flaky connectivity, and remote locations with no on-site IT staff.
Three design points follow:
- Local durability first. Write locally to persistent volumes, then replicate upstream when connectivity resumes.
- Backup connectivity. A second path is valuable if the first circuit goes down. If your sites do not have a redundant wired connection, one option to consider is a cellular IoT SIM backup for industrial routers. Check coverage and roaming agreements where your sites are located.
- Small replica counts. Two-node edge clusters can never maintain quorum during a failure. Either plan for three members, or accept a primary and warm standby with a well-documented manual procedure.
If your sites are datacentres, our enterprise edge and critical infrastructure pages describe the environments we engineer.
Monitoring stateful workloads
Pod count means very little here. You can see three of three pods up in a StatefulSet while replicas are hours behind the primary. Monitor the things that reflect actual data health:
- Readiness, tied to real health checks, not just an open port.
- Storage usage and growth rate, with alerts well before the volume fills.
- Restart count, because a crash loop on a database can corrupt data.
- Replication lag, often the earliest sign of a failing replica.
- Backup age and last successful restore test.
Alert on these to the same channel you use for the facility layer. In infrastructure terms, a database server reporting hot and a cooling loop trending out of range are often two sides of the same incident. If you are unfamiliar with Kubernetes in datacentre operations, refer to our Kubernetes glossary definition.
Our perspective
We make software run where failure incurs costs beyond dollars: datacentres, financial transaction systems, building telemetry and construction safety. Across more than 250 projects delivered, one lesson keeps recurring: stateful systems usually fail at the seams between components.
The StatefulSet works. The storage class works. The operator works. The failure happens where the volume zone, scheduling rule and backup job were configured by three different people who never tested them together. So we treat a stateful platform as one design, reviewed end to end, and we insist on a failure drill before go-live: drain a node, lose a zone, restore from backup, and time each one.
We also push teams to be candid about what should be running on the cluster itself. If the data is mission-critical and the team is small, a simpler architecture with fewer moving parts will often beat something more clever. If portability and operational control matter to your product, operators on Kubernetes are a strong choice, so long as day-two automation is designed in from the beginning rather than bolted on after the first major outage.
Our own experience includes datacentre platforms, telemetry pipelines, legacy modernisation, and AI or machine learning systems. All depend on durable data. The unsung hero layer between failure and success is usually the boring part: lifecycle automation, observability and proven recoverability.
Stateful platforms do not fail because one component is bad. They fail because the interactions between storage, scheduling, failover and recovery were never tested as one system.
— Harry
Where PODTECH can help
Moving a database, telemetry store or analytics pipeline to Kubernetes, or just making sense of one already there? We can provide engineers as a dedicated team for your project or augment your own. Common examples include legacy software modernisation for systems that predate containers, enterprise automation software for day-two operations after go-live, and SaaS development for multi-tenant platforms with complex data needs.
Planning a migration? Start with our database migration blueprints, and see our multi-tenant isolation patterns to learn how we keep customer data separate. Want to talk about a project? Get in touch at podtech.com.
A pre-production checklist
Before you call a stateful workload production-ready, confirm each item:
- The workload is deployed as a StatefulSet or controlled by an operator that provisions one, along with a headless Service.
- Each replica has its own persistent volume from a deliberately chosen storage class.
- Volume binding is delayed until scheduling and volume expansion is allowed.
- Spread constraints distribute replicas across nodes and zones.
- The PVC retention policy matches the data’s value.
- Backups execute on schedule, and restoring into a clean namespace has been timed.
- Failover has been triggered on purpose and the recovery time recorded.
- Alerts cover lag, storage, restarts and backup age.
- Database upgrades, operator upgrades and Kubernetes upgrades have been tested in staging.
- A named person owns each runbook.
FAQ
Is Kubernetes ready for stateful workloads?
Yes, but only for organisations that put effort into the ecosystem around it. Core capabilities like StatefulSets, persistent volumes and reliable operators satisfy most requirements. The CNCF survey found that most responding organisations were already running stateful applications in production. How ready you are depends more on your storage, backup and failover procedures than on Kubernetes itself.
What is the difference between a Deployment and a StatefulSet?
Deployments treat pods as fungible. StatefulSets assign each pod a stable name, its own persistent volume and an ordered, graceful lifecycle, as described in the Kubernetes docs. Deployments are a good fit for stateless services. StatefulSets are the right fit when identity or storage-per-replica is required.
Do I need an operator to run a database on Kubernetes?
You do not need one to start a database, but you usually do need one to run it safely. A StatefulSet will not handle failover, schema migrations or restores. The Data on Kubernetes 2022 report identified day-two automation as the biggest challenge, which is why operators have become the default solution.
What happens to the data when I delete a StatefulSet?
By default, PersistentVolumeClaims remain, so your data persists. You can adjust this with the PVC retention policy, which distinguishes between deletion and scale down. Leave the default in place for important data, and make sure you are comfortable with your backups before changing it.