Host disk will fill in 24h

What does this alert mean?

The current disk fill rate predicts the filesystem will be exhausted within 24 hours.

Contributing factors

• Log files growing unchecked (logrotate not configured or failing)
• Database WAL accumulation from stuck replication slots
• Large temporary files from batch jobs
• Core dumps filling /var

Affected components

• Any service writing to this filesystem
• The OS itself if / fills up

Expected action

Identify what is growing and either truncate/remove the files or provision additional disk space.

Metric

Description & unit

Linear projection of available filesystem bytes over the next 4 days, based on the 3-day trend.

Unit: bytes (alert fires when projected value < 0)

Threshold rationale

A 4-day forward projection gives enough lead time to act before the weekend or a public holiday.

Normal behavior

Available bytes decrease slowly and predictably. Sudden drops indicate runaway writes.

Screenshots

Silencing

Silence during planned disk migrations.

Expected frequency

Occasional — often precedes a disk-full incident by 1–3 days.

Severity guidance

Warning initially. Escalate to critical if disk is already above 90% full.

Impacted users

Users of all services writing to this filesystem.

Things to check to determine severity

• What filesystem is this? / vs /var vs /data
• What is growing? Logs, database files, tmp?

Prometheus query

Dashboards

• Node Disk — Fill in: dashboard URL

Steps

1. 1Find the largest directories: du -sh /* 2>/dev/null | sort -rh | head -20
2. 2Check log directory sizes: du -sh /var/log/*
3. 3Check if logrotate is working: systemctl status logrotate and ls -lht /var/log/*.gz | head
4. 4Find recently modified large files: find / -size +100M -mtime -1 2>/dev/null
5. 5Check for core dumps: ls -lh /var/crash/ /tmp/*.core 2>/dev/null
6. 6If PostgreSQL WAL: check for stuck replication slots: SELECT slot_name, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) FROM pg_replication_slots;

Useful commands

Guidance

Escalate if current disk usage is above 90% — do not wait for the prediction window.

Slack channels

• #infrastructure-oncall — Fill in: channel URL

Alert rule

Tuning notes

The predict_linear window of 3d smooths out daily cycles. Do not reduce below 1d or transient spikes will cause false positives.