Skip to main content
SRE

Prometheus

Prometheus target missing

warning

Overview

What does this alert mean?

A scrape target has disappeared from the active targets list. The up series for this target is absent.

Contributing factors

  • Service restarted and changed IP or port
  • Service discovery configuration changed
  • Target is down or unreachable on the metrics port
  • Relabeling rule dropped the target

Affected components

  • Alerting rules that depend on metrics from this target
  • Dashboards showing metrics from this target

Expected action

Check the Prometheus targets UI for the last error, then restore the target or update the scrape config.

Services

Metrics

Metric

up

Description & unit

Synthetic gauge written by Prometheus after each scrape. 1 = scrape succeeded, 0 = failed.

Unit: boolean (0 or 1)

Threshold rationale

A missing target means a gap in metric coverage. Alerting rules relying on those metrics become unreliable.

Normal behavior

All configured targets show up=1 after each scrape interval.

Screenshots

Prometheus targets UI showing a down or missing target
Prometheus targets UI showing a down or missing target
Target back in UP state
Target back in UP state

Alert Behavior

Silencing

Silence when decommissioning a target intentionally. Remove the target from the scrape config instead of silencing long-term.

Expected frequency

Occasional — often follows a service restart, deploy, or network change.

Severities

Severity guidance

Warning. Escalate if the missing target is a critical service with no redundancy.

Impacted users

SRE team — missing targets create blind spots in alerting coverage.

Things to check to determine severity

  • Is the target service running?
  • Did the IP or port change?
  • Is a relabeling rule dropping it?

Verification

Prometheus query

absent(up{job='<job-name>'})

Dashboards

  • Prometheus Targets — Fill in: dashboard URL

Log queries

  • Prometheus logs — Fill in: log query URL

Troubleshooting

Steps

  1. 1Open the Prometheus targets UI: http://:9090/targets
  2. 2Filter by the job name — read the last error message
  3. 3From the Prometheus host, curl the metrics endpoint directly: curl http://:/metrics | head
  4. 4If using Kubernetes SD: verify the pod/service labels match the relabeling rules in the scrape config
  5. 5If using static config: verify the host and port are correct and reachable
  6. 6Check if a recent relabeling change dropped the target: http://:9090/config

Useful commands

curl -s http://<target-host>:<port>/metrics | head -20curl -s http://<prometheus-host>:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up")'

Possible Resolutions

  • Service was restarted and the pod IP changed. Kubernetes SD picked it up within one scrape interval after the pod became Ready.

Dependencies

  • Service discovery mechanism (Kubernetes SD, Consul SD, static config)
  • Network connectivity between Prometheus and target

Escalation

Guidance

Escalate if a critical service's metrics have been missing for more than one scrape interval and the cause is unknown.

Slack channels

  • #observability-oncall — Fill in: channel URL

Definitions

Alert rule

Fill in: link to the alert rule in _data/rules.yml on GitHub

Tuning notes

Add `for: 5m` to avoid alerting on targets that briefly disappear during rolling restarts.

Related Links

Template adapted from https://runbooks.gitlab.com/template-alert-playbook/