Skip to main content
SRE

Kubernetes

Kubernetes node not ready

critical

Overview

What does this alert mean?

A node is in NotReady state. Pods scheduled on it may be evicted or unreachable.

Contributing factors

  • Kubelet crashed or is not running
  • Node is out of disk, memory, or PID resources
  • Network partition — node cannot reach the API server

Affected components

  • All pods scheduled on the affected node
  • Services with endpoints on this node

Expected action

SSH to the node, check kubelet status, and restore the node to Ready state. If unrecoverable, cordon and drain the node.

Services

Metrics

Metric

kube_node_status_condition{condition='Ready',status='true'}

Description & unit

Gauge that is 1 when the node's Ready condition is True, 0 otherwise.

Unit: boolean (0 or 1)

Threshold rationale

Any node in NotReady state should be investigated immediately.

Normal behavior

All nodes should have Ready=true at all times.

Screenshots

Dashboard when node NotReady alert fires
Dashboard when node NotReady alert fires
Dashboard after node returns to Ready
Dashboard after node returns to Ready

Alert Behavior

Silencing

Silence during planned node maintenance with `kubectl cordon` first.

Expected frequency

Uncommon — usually indicates a hardware failure or kubelet crash.

Severities

Severity guidance

Critical. A NotReady node may cause pod evictions and service disruption.

Impacted users

Users of services with pods on the affected node.

Things to check to determine severity

  • How many nodes are affected?
  • Are there pending pods that cannot be rescheduled?
  • Is the control plane healthy?

Verification

Prometheus query

kube_node_status_condition{condition='Ready',status='true'} == 0

Dashboards

  • Kubernetes Nodes — Fill in: dashboard URL

Log queries

  • Kubelet logs — Fill in: log query URL

Troubleshooting

Steps

  1. 1Check all node statuses: kubectl get nodes -o wide
  2. 2Describe the affected node: kubectl describe node
  3. 3SSH to the node and check kubelet: systemctl status kubelet
  4. 4Check kubelet logs: journalctl -u kubelet --since '10 minutes ago' | tail -50
  5. 5Check node conditions for resource pressure: kubectl describe node | grep -A5 Conditions
  6. 6If kubelet is stopped: systemctl restart kubelet
  7. 7If node is unrecoverable: kubectl cordon && kubectl drain --ignore-daemonsets --delete-emptydir-data

Useful commands

kubectl get nodes -o widekubectl describe node <node-name>systemctl status kubeletjournalctl -u kubelet --since '10 minutes ago' | tail -50

Possible Resolutions

  • Disk pressure caused kubelet to mark node NotReady. Clearing unused images with `crictl rmi --prune` resolved it.

Dependencies

  • Container runtime (containerd/CRI-O)
  • Cloud provider node health checks
  • etcd (for API server connectivity)

Escalation

Guidance

Escalate if more than one node is NotReady or if pods cannot be rescheduled.

Slack channels

  • #platform-oncall — Fill in: channel URL

Definitions

Alert rule

Fill in: link to the alert rule in _data/rules.yml on GitHub

Tuning notes

Add `for: 2m` to avoid alerting on transient API server connectivity blips.

Related Links

Template adapted from https://runbooks.gitlab.com/template-alert-playbook/