Skip to main content
SRE

PostgreSQL

PostgreSQL down

critical

Overview

What does this alert mean?

pg_up == 0. The PostgreSQL instance is unreachable.

Contributing factors

  • Postmaster process crashed
  • Disk full — WAL directory or data directory
  • OOM kill

Affected components

  • All services reading or writing to this PostgreSQL instance

Expected action

Immediately investigate and restore the PostgreSQL process.

Services

Metrics

Metric

pg_up

Description & unit

Binary gauge from postgres_exporter. 1 = reachable, 0 = down.

Unit: boolean (0 or 1)

Threshold rationale

Any value of 0 means the instance is unreachable.

Normal behavior

Always 1 during normal operation.

Screenshots

Dashboard when pg_up == 0
Dashboard when pg_up == 0
Dashboard after recovery
Dashboard after recovery

Alert Behavior

Silencing

Silence during planned maintenance windows.

Expected frequency

Rare.

Severities

Severity guidance

Critical. If primary is down, assess whether replicas can serve reads and whether writes are blocked.

Impacted users

All users of dependent services.

Things to check to determine severity

  • Is this primary or replica?
  • Is disk full?
  • Are OOM kills visible in dmesg?

Verification

Prometheus query

pg_up == 0

Dashboards

  • PostgreSQL Overview — Fill in: dashboard URL

Log queries

  • PostgreSQL logs — Fill in: log query URL

Troubleshooting

Steps

  1. 1Check PostgreSQL logs: journalctl -u postgresql --since '5 minutes ago'
  2. 2Check process status: systemctl status postgresql
  3. 3Check disk: df -h /var/lib/postgresql
  4. 4Check WAL directory size: du -sh /var/lib/postgresql/*/pg_wal
  5. 5Attempt restart: systemctl restart postgresql

Useful commands

systemctl status postgresqljournalctl -u postgresql --since '10 minutes ago'df -h /var/lib/postgresql

Possible Resolutions

  • OOM kill due to work_mem set too high. Reducing work_mem resolved it.

Dependencies

  • Application services
  • postgres_exporter

Escalation

Guidance

Escalate immediately if primary is down.

Slack channels

  • #database-oncall — Fill in: channel URL

Definitions

Alert rule

Fill in: link to the alert rule in _data/rules.yml on GitHub

Tuning notes

Add `for: 1m` to tolerate restarts without paging.

Related Links

Template adapted from https://runbooks.gitlab.com/template-alert-playbook/