Skip to main content
SRE

Host & Hardware

Host out of memory

critical

Overview

What does this alert mean?

Available memory is less than 10% of total. OOM kills are imminent or already occurring.

Contributing factors

  • Memory leak in an application process
  • Unexpected traffic spike exhausting application caches
  • Runaway process consuming unbounded memory

Affected components

  • All processes on the host — the kernel OOM killer may terminate any of them
  • Swap (if configured) — high swap usage causes severe I/O latency

Expected action

Identify the top memory consumers and either kill the runaway process or migrate services off the host.

Services

Metrics

Metric

node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

Description & unit

Ratio of available memory (including reclaimable cache) to total physical memory.

Unit: ratio (0–1), alert fires when < 0.1

Threshold rationale

Below 10% available, the kernel OOM killer becomes active.

Normal behavior

Typically 30–60% available on a healthy host.

Screenshots

Dashboard when available memory drops below 10%
Dashboard when available memory drops below 10%
Dashboard after memory recovers
Dashboard after memory recovers

Alert Behavior

Silencing

Silence during known memory-intensive batch jobs with a time-bounded silence.

Expected frequency

Uncommon — usually indicates a memory leak or misconfiguration.

Severities

Severity guidance

Critical. OOM kills can take down any process on the host without warning.

Impacted users

All users of services running on this host.

Things to check to determine severity

  • Has the kernel already OOM-killed any processes? (`dmesg | grep -i 'killed process'`)
  • Is swap enabled and exhausted?
  • Which process is growing?

Verification

Prometheus query

node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1

Dashboards

  • Node Overview — Fill in: dashboard URL

Log queries

  • OOM kill events — Fill in: log query URL

Troubleshooting

Steps

  1. 1Identify top memory consumers: ps aux --sort=-%mem | head -20
  2. 2Check OOM kill history: dmesg | grep -i 'killed process'
  3. 3Check swap usage: free -h
  4. 4If a specific service is leaking, restart it: systemctl restart
  5. 5If the leak is persistent, capture a heap dump before restarting for post-mortem analysis

Useful commands

ps aux --sort=-%mem | head -20dmesg | grep -i 'killed process'free -hvmstat -s | head -10

Possible Resolutions

  • Memory leak in a Java service. Restarting the service recovered the host; heap profiling identified the leak.

Dependencies

  • All services running on the host
  • OOM killer configuration (vm.oom_score_adj)

Escalation

Guidance

Escalate if OOM kills have already occurred or if the host is unresponsive.

Slack channels

  • #infrastructure-oncall — Fill in: channel URL

Definitions

Alert rule

Fill in: link to the alert rule in _data/rules.yml on GitHub

Tuning notes

The 10% threshold is a ratio of total memory — self-normalizing across host sizes.

Related Links

Template adapted from https://runbooks.gitlab.com/template-alert-playbook/