Skip to main content
SRE

Kubernetes

Kubernetes pod crash looping

warning

Overview

What does this alert mean?

A pod's restart count is high. The container is repeatedly crashing and being restarted by kubelet.

Contributing factors

  • Application crash — unhandled exception, OOM kill, or bad CMD/entrypoint
  • Misconfigured liveness probe killing a healthy-but-slow-starting container
  • Missing ConfigMap, Secret, or volume mount
  • Failing init containers preventing the main container from starting
  • App starts before a dependency (DB, cache) is ready — no retry/backoff logic
  • File permission issues on mounted volumes
  • SIGTERM not handled — container exits non-zero on graceful shutdown

Affected components

  • The specific pod and the service endpoints it provides

Expected action

Check the pod's previous container logs and exit code to identify the crash reason, then fix the root cause. Do not restart or delete the pod before gathering evidence.

Services

Metrics

Metric

kube_pod_container_status_restarts_total

Description & unit

Cumulative restart count per container. Alert fires when the rate of restarts is high.

Unit: restarts per 15 minutes

Threshold rationale

More than 3 restarts in 15 minutes indicates a persistent crash loop, not a transient startup failure.

Normal behavior

Restart count is stable. A single restart on deploy is acceptable.

Screenshots

Dashboard when crash loop alert fires
Dashboard when crash loop alert fires
Dashboard after pod stabilizes
Dashboard after pod stabilizes

Alert Behavior

Silencing

Silence during deploys if the new version is expected to fail fast during rollout.

Expected frequency

Occasional — often accompanies a bad deploy or config change.

Severities

Severity guidance

Warning. Escalate to critical if the crash-looping pod is a singleton (no replicas) serving production traffic.

Impacted users

Users of the service the pod belongs to.

Things to check to determine severity

  • Is the pod part of a Deployment with multiple replicas?
  • Was there a recent deploy or config change?
  • Is the crash an OOM kill or an application exception?
  • Are resource limits (CPU/memory) too tight for the workload?
  • Are readiness or liveness probes configured correctly?
  • Are all required ConfigMaps and Secrets present and mounted?
  • Are downstream dependencies (databases, external APIs) reachable from the pod?
  • Is the node the pod is scheduled on healthy?

Verification

Prometheus query

increase(kube_pod_container_status_restarts_total[15m]) > 3

Dashboards

  • Kubernetes Pods — Fill in: dashboard URL

Log queries

  • Pod logs — Fill in: log query URL

Troubleshooting

Steps

  1. 1Find crash-looping pods: kubectl get pods -A | grep -v Running | grep -v Completed
  2. 2Describe the pod for events and failure reasons: kubectl describe pod -n
  3. 3Check previous container logs (what happened right before the crash): kubectl logs -n --previous
  4. 4Get the exit code of the last terminated container: kubectl get pod -n -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
  5. 5Check cluster-wide events for context: kubectl get events -n --sort-by='.lastTimestamp'
  6. 6Check node health — a failing node can cause pods to crash: kubectl get nodes
  7. 7If OOM (exit code 137): check memory limits in the pod spec and increase if appropriate
  8. 8If liveness probe is killing the container: check initialDelaySeconds and failureThreshold — increase if the app starts slowly
  9. 9If missing config: look for MountVolume or secret not found errors in kubectl describe Events
  10. 10If the crash is hard to reproduce, run the container interactively to inspect it: kubectl run debug --image= -it --restart=Never -- bash
  11. 11To stop the restart loop while debugging, set restartPolicy: Never on a test pod so Kubernetes doesn't mask the failure with retries

Useful commands

kubectl get pods -A | grep -v Running | grep -v Completedkubectl describe pod <pod-name> -n <namespace>kubectl logs <pod-name> -n <namespace> --previouskubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'kubectl get events -n <namespace> --sort-by='.lastTimestamp'kubectl get nodeskubectl run debug --image=<image> -it --restart=Never -- bash

Possible Resolutions

  • liveness probe initialDelaySeconds was too low for a slow-starting JVM app. Increasing it stopped the crash loop.

  • Bad CMD in the Docker image — entrypoint script exited with code 1 immediately. Identified via kubectl logs --previous and fixed in the Dockerfile.

  • App crashed on startup because the database was not yet ready. Adding an init container with a readiness check resolved the ordering issue.

Dependencies

  • ConfigMaps and Secrets the pod mounts
  • Downstream services the pod connects to on startup

Escalation

Guidance

Escalate if the crash-looping pod is a singleton or if the crash is causing data corruption.

Slack channels

  • #platform-oncall — Fill in: channel URL

Definitions

Alert rule

Fill in: link to the alert rule in _data/rules.yml on GitHub

Tuning notes

The threshold of 3 restarts in 15 minutes distinguishes crash loops from normal rolling restarts on deploy.

Related Links

Template adapted from https://runbooks.gitlab.com/template-alert-playbook/