Kubernetes node memory pressure

What does this alert mean?

A node's available memory has fallen below the kubelet eviction threshold. The node is set to MemoryPressure=True, new pods are blocked from scheduling via the node.kubernetes.io/memory-pressure:NoSchedule taint, and running pods are being evicted to reclaim memory.

Contributing factors

• Workloads have no memory requests — BestEffort pods are scheduled without capacity reservation and are first to be evicted
• Memory requests are set too low relative to actual usage — scheduler overcommits the node
• Memory leak in an application — pod restart count climbs with OOMKilled reason
• Active page cache inflating kubelet working-set calculation — kubelet counts active file-backed pages, which can diverge from free -m output
• Eviction threshold set too tight for the node size — normal load trips the threshold
• Thundering herd or bursty workloads — sudden spike in memory consumption

Affected components

• All pods on the pressure node — BestEffort evicted first, then over-request Burstable, then Guaranteed/within-request Burstable last
• Pods pending scheduling on the affected node (blocked by NoSchedule taint)
• Downstream services depending on evicted pods

Expected action

Confirm the node condition and taint, distinguish kubelet eviction from kernel OOMKill, identify the highest memory consumers, and either right-size requests/limits or cordon the node and redistribute workloads.

Metric

Description & unit

Gauge that is 1 when the node's MemoryPressure condition is True.

Threshold rationale

Fill in: why this threshold value was chosen

Normal behavior

Fill in: what this metric looks like when healthy

Silencing

Fill in: when and how to silence this alert

Expected frequency

Fill in: expected firing frequency

Severity guidance

Fill in: how to assign incident severity for this alert

Impacted users

Fill in: who is affected when this fires

Prometheus query

Steps

1. 1Confirm MemoryPressure condition and taint: kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="MemoryPressure")].status}{"\n"}{end}'
2. 2Distinguish kubelet eviction from kernel OOMKill: inspect lastState.terminated.reason — OOMKilled means the container hit its cgroup memory limit; Evicted means kubelet removed the pod due to node pressure. OOMKilled points to a limit or leak; Evicted points to node overcommit.
3. 3Find evicted pods: kubectl get pods --all-namespaces --field-selector status.phase=Failed | grep Evicted
4. 4Check current memory consumers: kubectl top pods -n --sort-by=memory
5. 5Compare allocated vs. actual memory on the node: kubectl describe node | grep -A10 'Allocated resources'
6. 6Check actual node memory: kubectl get --raw /api/v1/nodes//proxy/metrics | grep kubelet_evictions
7. 7Inspect kubelet eviction configuration: grep -A5 'evictionHard' /var/lib/kubelet/config.yaml
8. 8Check pod QoS classes to predict eviction risk: kubectl get pod -n -o jsonpath='{.status.qosClass}'
9. 9If a memory leak is suspected, check restart count and last termination: kubectl describe pod -n | grep -A5 'Last State'

Useful commands

Guidance

Escalate if Guaranteed QoS pods are being evicted (indicates severe node oversubscription), if MemoryPressure persists after eviction, or if multiple nodes enter pressure simultaneously.

Slack channels

• #platform-oncall — Fill in: channel URL

Alert rule

Tuning notes

Set memory requests to steady-state usage and limits with headroom for spikes. Use LimitRange to enforce defaults and prevent BestEffort pods. Configure --kube-reserved and --system-reserved so the scheduler accounts for node overhead. Tune --eviction-minimum-reclaim to avoid rapid condition flapping.