Kubernetes container OOM killed

What does this alert mean?

A container was terminated by the Linux kernel's OOM killer because it exceeded its cgroup memory limit. Exit code 137 (SIGKILL) is recorded with Reason: OOMKilled.

Contributing factors

• Memory limit set below actual peak usage — sized from averages or staging, not production peaks
• Memory leak — usage grows steadily across container lifetimes until it hits the limit
• Traffic or workload spike — large payloads, fan-out queries, or batch jobs push usage past the steady-state baseline
• JVM/runtime heap misconfigured — heap ceiling set above the container limit, so the cgroup kills before the runtime throws OutOfMemoryError
• Node memory pressure — kubelet evicts pods when the node runs low, before individual containers hit their own limits

Affected components

• The OOMKilled container and the pod it belongs to
• In-flight requests and open connections (SIGKILL gives no time to flush or close)
• Downstream services that lose the connection without a graceful shutdown

Expected action

Confirm OOMKilled is the cause (not a probe failure or eviction), identify whether the root cause is a bad limit, a leak, or a spike, then apply the matching fix.

Metric

Description & unit

Gauge exported by kube-state-metrics. Value is 1 when the last termination reason matches the label value (e.g. OOMKilled).

Unit: boolean (0 or 1)

Threshold rationale

Any OOMKilled termination is actionable — either the limit is wrong or the application has a leak.

Normal behavior

Containers should never be OOMKilled. A single occurrence warrants investigation; repeated occurrences indicate a persistent misconfiguration or leak.

Screenshots

Silencing

Do not silence. OOMKilled events are always worth investigating. If a planned batch job is expected to use high memory, increase its limit before running, not after.

Expected frequency

Should be rare. Repeated OOMKills on the same workload indicate the root cause has not been fixed.

Severity guidance

Critical. SIGKILL is abrupt — in-flight work is lost and connections drop without warning. Stateful workloads risk data corruption.

Impacted users

Users of the service the pod belongs to, for the duration of the restart.

Things to check to determine severity

• Is the exit code 137 with Reason: OOMKilled, or is it a probe failure or eviction? (Check kubectl describe pod)
• Is this a single container repeatedly hitting its limit, or multiple pods dying together? (Multiple = node pressure)
• Does memory usage grow steadily over time (leak), spike at specific events (bad limit or spike), or is this the first occurrence?
• What is the container's memory limit vs. its actual peak usage?
• Is the node showing MemoryPressure?

Prometheus query

Dashboards

• Kubernetes Pods — Fill in: dashboard URL

Log queries

• Pod logs (previous container) — Fill in: log query URL

Steps

1. 1Confirm OOMKilled is the cause — look for Reason: OOMKilled and Exit Code: 137 in Last State: kubectl describe pod -n
2. 2Check if this is node pressure (multiple pods affected) vs. a single container hitting its own limit: kubectl describe node | grep -A5 MemoryPressure
3. 3Check current memory usage vs. limits: kubectl top pod -n
4. 4Check the configured limit: kubectl get pod -n -o jsonpath='{.spec.containers[0].resources.limits.memory}'
5. 5Read previous container logs for allocation failures, GC thrash, or heap errors before the kill: kubectl logs -n --previous
6. 6Chart memory over time in Prometheus to distinguish a leak (steady climb across restarts) from a bad limit (flat baseline, spike at events): container_memory_working_set_bytes{pod=''} / container_spec_memory_limit_bytes{pod=''}
7. 7If a memory leak is suspected, exec into the container and check process memory: kubectl exec -it -n -- cat /proc/1/status | grep -i VmRSS
8. 8If the container uses a JVM, confirm heap is sized below the container limit (e.g. -Xmx at ~75% of the limit) to leave room for off-heap memory
9. 9As a quick fix while investigating, patch the memory limit: kubectl patch deployment -n -p '{"spec":{"template":{"spec":{"containers":[{"name":"","resources":{"limits":{"memory":"1Gi"}}}]}}}}'

Useful commands

Guidance

Escalate if the OOMKilled pod is a singleton, a stateful workload with risk of data corruption, or if node MemoryPressure is causing multiple pods to die together.

Slack channels

• #platform-oncall — Fill in: channel URL

Alert rule

Tuning notes

Set memory limits at p99 of production peak usage, not average. Keep requests-to-limits ratio below 4x to avoid implicit node overcommitment. For Guaranteed QoS (last evicted under node pressure), set requests equal to limits.