Skip to main content
SRE

MySQL

MySQL down

critical

Overview

What does this alert mean?

The mysqld_exporter reports mysql_up == 0. The MySQL instance is unreachable.

Contributing factors

  • Process crashed or was OOM-killed
  • Disk full — InnoDB cannot write redo logs
  • Corrupted data directory
  • Authentication plugin change broke the exporter user

Affected components

  • All application services that read or write to this MySQL instance
  • mysqld_exporter scrape target

Expected action

Immediately investigate and restore the MySQL process. If this is the primary, check whether replicas can serve read traffic in the interim.

Services

Metrics

Metric

mysql_up

Description & unit

Binary gauge exported by mysqld_exporter. Value is 1 when the exporter can connect to MySQL, 0 when it cannot.

Unit: boolean (0 or 1)

Threshold rationale

Any value of 0 means the instance is completely unreachable. There is no partial state.

Normal behavior

Should always be 1. A value of 0 is never expected during normal operation.

Screenshots

Dashboard when mysql_up == 0 is firing
Dashboard when mysql_up == 0 is firing
Dashboard after MySQL recovers and mysql_up returns to 1
Dashboard after MySQL recovers and mysql_up returns to 1

Alert Behavior

Silencing

Silence via Alertmanager during planned maintenance windows. Use a time-bounded silence with a comment explaining the maintenance.

Expected frequency

Rare — should only fire on process crash, OOM kill, or network partition.

Severities

Severity guidance

This is critical severity. If the primary is down and no replica can serve writes, treat as P1.

Impacted users

All users of services that depend on this MySQL instance.

Things to check to determine severity

  • Is this the primary or a read replica?
  • Are application services logging connection errors?
  • Is the disk full on the MySQL host?

Verification

Prometheus query

mysql_up == 0

Dashboards

  • MySQL Overview — Fill in: dashboard URL

Log queries

  • MySQL error logs — Fill in: log query URL

Troubleshooting

Steps

  1. 1Check the MySQL error log: journalctl -u mysql --since '5 minutes ago' or /var/log/mysql/error.log
  2. 2Check if the process is running: systemctl status mysql
  3. 3Check disk space: df -h /var/lib/mysql
  4. 4Attempt to start: systemctl start mysql and capture the output
  5. 5If disk full, find the largest files: du -sh /var/lib/mysql/* | sort -rh | head
  6. 6Verify exporter connectivity: mysql -u exporter -p -h localhost -e 'SELECT 1'

Useful commands

systemctl status mysqljournalctl -u mysql --since '10 minutes ago'df -h /var/lib/mysqlmysql -u exporter -p -h localhost -e 'SELECT 1'

Possible Resolutions

  • Process was OOM-killed. Increasing memory limits and adding a swap file resolved it.

  • Disk full due to unchecked binary log growth. Purging old binlogs restored service.

Dependencies

  • Application services (read/write traffic)
  • Backup agent
  • mysqld_exporter

Escalation

Guidance

Escalate immediately if the primary is down and replicas cannot serve write traffic. Page the database team.

Slack channels

  • #database-oncall — Fill in: channel URL

Definitions

Alert rule

Fill in: link to the alert rule in _data/rules.yml on GitHub

Tuning notes

Threshold of 0 is non-negotiable — any value of 0 is a real outage. The `for: 1m` delay on the alert rule tolerates a single restart.

Related Links

Template adapted from https://runbooks.gitlab.com/template-alert-playbook/