Kubernetes StatefulSet down

What does this alert mean?

A StatefulSet has fewer ready replicas than desired. One or more pods are not running or passing readiness checks.

Contributing factors

• PVC stuck in Pending — storage class misconfiguration or provisioner failure
• Orphaned PV from a previous release — reclaim policy left the volume in Released state
• Pod stuck in Terminating — volume detach stalled after a node failure or upgrade
• Volume mount conflict — overlapping mount paths or subPath directory missing
• File permission issue — container cannot write to the mounted volume
• Headless service selector mismatch — pods start but are unreachable by ordinal DNS name
• Node cordoned or tainted after a cluster upgrade — pods cannot reschedule
• Ordered rollout blocked — pod-N cannot start until pod-(N-1) is Ready

Affected components

• The StatefulSet and any service that routes to its pods
• Dependent services relying on stable pod DNS names (e.g. <pod>.<headless-svc>.<ns>.svc.cluster.local)

Expected action

Check StatefulSet replica status and the lowest-ordinal failing pod first — ordered startup means one stuck pod blocks all higher-ordinal pods.

Metric

Description & unit

Ratio of ready replicas to desired replicas for the StatefulSet. Exported by kube-state-metrics.

Unit: ratio (0–1)

Threshold rationale

Any value below 1.0 means at least one replica is not serving traffic. For quorum-based systems (etcd, ZooKeeper, Kafka), losing a majority of replicas is a P1 outage.

Normal behavior

readyReplicas == replicas at all times outside of a rolling update.

Screenshots

Silencing

Silence during planned rolling upgrades. Use a time-bounded Alertmanager silence scoped to the StatefulSet name.

Expected frequency

Uncommon in steady state — typically follows a cluster upgrade, storage change, or bad deploy.

Severity guidance

Critical if replicas drop below quorum for consensus systems (etcd, Kafka, ZooKeeper, Cassandra). Warning if at least one replica is healthy and the service can still serve traffic.

Impacted users

Users of any service backed by this StatefulSet.

Things to check to determine severity

• Which ordinal pod is failing first? (pod-0 blocks pod-1, pod-1 blocks pod-2)
• Are the PVCs Bound or stuck Pending?
• Did a cluster upgrade change the storage class or node availability zone?
• Is the headless service selector still matching the pod labels?
• Are any nodes cordoned that the StatefulSet pods were scheduled on?

Prometheus query

Dashboards

• Kubernetes StatefulSets — Fill in: dashboard URL

Log queries

• Pod logs for failing ordinal — Fill in: log query URL

Steps

1. 1Check StatefulSet replica status — note READY vs desired: kubectl get statefulsets -n
2. 2Get detailed events and rollout strategy: kubectl describe statefulset -n
3. 3Check if the rollout is stuck mid-update: kubectl rollout status statefulset -n
4. 4Identify which pod is failing — start from the lowest ordinal: kubectl get pods -n -o wide
5. 5Describe the failing pod for mount errors, scheduling failures, or probe failures: kubectl describe pod -0 -n
6. 6Check PVC status — Pending means storage has not been provisioned: kubectl get pvc -n
7. 7If a PVC is Pending, describe it to see the provisioner error: kubectl describe pvc -n
8. 8Check for orphaned PVs stuck in Released state: kubectl get pv | grep Released
9. 9Verify the storage class exists and the provisioner is healthy: kubectl get storageclass && kubectl get pods -n kube-system | grep provisioner
10. 10Confirm the headless service exists and its selector matches pod labels: kubectl get svc -n && kubectl get endpoints -n
11. 11Check recent events sorted by time: kubectl get events -n --sort-by=.metadata.creationTimestamp
12. 12If a pod is stuck Terminating after a node failure, force-delete it — the StatefulSet controller will recreate it: kubectl delete pod -0 --force --grace-period=0 -n
13. 13To roll back to a previous revision: kubectl rollout history statefulset -n then kubectl rollout undo statefulset -n

Useful commands

Guidance

Escalate immediately if the StatefulSet is a quorum-based system (etcd, Kafka, ZooKeeper) and ready replicas drop below majority. Data loss is possible if the remaining pods are restarted before the failed one recovers.

Slack channels

• #platform-oncall — Fill in: channel URL

Alert rule

Tuning notes

For single-replica StatefulSets, any pod failure fires immediately (0/1 = 0). For multi-replica sets, consider alerting on < 1 (any degradation) vs. < 0.5 (quorum loss) depending on tolerance.