Prometheus rule evaluation slow

What does this alert mean?

A rule group's evaluation takes longer than its configured interval. Alerts from this group may be delayed.

Contributing factors

• Complex PromQL with high label cardinality
• Prometheus under-resourced — insufficient CPU
• TSDB under heavy concurrent write load

Affected components

• All alerting and recording rules in the slow rule group
• Dependent dashboards that query the recording rules

Expected action

Identify the slow rule group, pre-compute expensive sub-expressions as recording rules, or reduce cardinality.

Metric

Description & unit

Duration of the last evaluation of a rule group, in seconds.

Unit: seconds

Threshold rationale

If evaluation takes longer than the group interval, subsequent evaluations queue up and alerts are delayed.

Normal behavior

Rule group evaluation should complete in well under the group interval (typically < 10% of the interval).

Screenshots

Silencing

Silence while actively optimizing the rule group.

Expected frequency

Rare — usually appears after cardinality growth or a new complex rule is added.

Severity guidance

Warning. Escalate if evaluation lag is causing known alert delays in production.

Impacted users

SRE team — delayed alerts mean slower incident detection.

Things to check to determine severity

• Which rule group is slow? (`prometheus_rule_group_last_duration_seconds`)
• What is the cardinality of the metrics involved?
• Was a new rule recently added?

Prometheus query

Dashboards

• Prometheus Self-Monitoring — Fill in: dashboard URL

Log queries

• Prometheus slow query logs — Fill in: log query URL

Steps

1. 1Identify slow groups: query prometheus_rule_group_last_duration_seconds and sort descending
2. 2In Prometheus UI go to Status → Rules — look for groups with high last duration
3. 3For each slow rule, run the PromQL manually and time it: note the cardinality of the result
4. 4Pre-compute expensive sub-expressions as recording rules with a shorter interval
5. 5Check Prometheus CPU usage — if saturated, scale vertically or shard rule groups

Useful commands

Guidance

Escalate if evaluation lag exceeds 2x the group interval for more than 10 minutes.

Slack channels

• #observability-oncall — Fill in: channel URL

Alert rule

Tuning notes

Split large rule groups into smaller ones with focused scopes to reduce per-group evaluation time.