PostgreSQL
PostgreSQL down
Overview
What does this alert mean?
pg_up == 0. The PostgreSQL instance is unreachable.
Contributing factors
- Postmaster process crashed
- Disk full — WAL directory or data directory
- OOM kill
Affected components
- All services reading or writing to this PostgreSQL instance
Expected action
Immediately investigate and restore the PostgreSQL process.
Services
- PostgreSQLhttps://www.postgresql.org/docs/Team: Fill in: owning team name
Metrics
Metric
pg_upDescription & unit
Binary gauge from postgres_exporter. 1 = reachable, 0 = down.
Unit: boolean (0 or 1)
Threshold rationale
Any value of 0 means the instance is unreachable.
Normal behavior
Always 1 during normal operation.
Screenshots


Alert Behavior
Silencing
Silence during planned maintenance windows.
Expected frequency
Rare.
Severities
Severity guidance
Critical. If primary is down, assess whether replicas can serve reads and whether writes are blocked.
Impacted users
All users of dependent services.
Things to check to determine severity
- Is this primary or replica?
- Is disk full?
- Are OOM kills visible in dmesg?
Verification
Prometheus query
pg_up == 0Dashboards
- PostgreSQL Overview — Fill in: dashboard URL
Log queries
- PostgreSQL logs — Fill in: log query URL
Troubleshooting
Steps
- 1Check PostgreSQL logs:
journalctl -u postgresql --since '5 minutes ago' - 2Check process status:
systemctl status postgresql - 3Check disk:
df -h /var/lib/postgresql - 4Check WAL directory size:
du -sh /var/lib/postgresql/*/pg_wal - 5Attempt restart:
systemctl restart postgresql
Useful commands
systemctl status postgresqljournalctl -u postgresql --since '10 minutes ago'df -h /var/lib/postgresqlPossible Resolutions
OOM kill due to work_mem set too high. Reducing work_mem resolved it.
Dependencies
- Application services
- postgres_exporter
Escalation
Guidance
Escalate immediately if primary is down.
Slack channels
- #database-oncall — Fill in: channel URL
Definitions
Alert rule
Fill in: link to the alert rule in _data/rules.yml on GitHubTuning notes
Add `for: 1m` to tolerate restarts without paging.
Related Links
Related alerts
Documentation
Template adapted from https://runbooks.gitlab.com/template-alert-playbook/