Skip to main content
SRE

Kubernetes

Kubernetes CronJob too long

warning

Overview

What does this alert mean?

A CronJob has been running longer than expected, or has stopped being scheduled entirely.

Contributing factors

  • No activeDeadlineSeconds set — a hung job runs indefinitely
  • concurrencyPolicy: Allow — previous runs pile up and block resources
  • Job stuck on network I/O, external dependency, or a deadlock
  • More than 100 missed start times — the controller stops scheduling silently
  • startingDeadlineSeconds set too low — start window closes before the job can launch
  • CronJob is suspended (spec.suspend: true)
  • UTC timezone confusion — schedule fires at unexpected local times if spec.timeZone is not set

Affected components

  • Batch jobs and scheduled tasks
  • Downstream systems depending on CronJob output (reports, syncs, backups)

Expected action

Identify whether the job is hung or not scheduling. Set activeDeadlineSeconds to cap run time, set concurrencyPolicy: Forbid to prevent overlap, and recreate the CronJob if more than 100 schedules were missed.

Services

Metrics

Metric

kube_cronjob_status_active

Description & unit

Number of currently active jobs for a CronJob. A long-running value indicates a stuck job.

Threshold rationale

Fill in: why this threshold value was chosen

Normal behavior

Fill in: what this metric looks like when healthy

Alert Behavior

Silencing

Fill in: when and how to silence this alert

Expected frequency

Fill in: expected firing frequency

Severities

Severity guidance

Fill in: how to assign incident severity for this alert

Impacted users

Fill in: who is affected when this fires

Verification

Prometheus query

Fill in: the PromQL query that triggered this alert

Troubleshooting

Steps

  1. 1Check CronJob status and last schedule time: kubectl get cronjob -n
  2. 2Inspect events and schedule history: kubectl describe cronjob -n — look at LAST SCHEDULE and any warning events about missed start times
  3. 3Check if the CronJob is suspended: kubectl get cronjob -n -o jsonpath='{.spec.suspend}'
  4. 4List all jobs created by the CronJob: kubectl get jobs -n -l job-name=
  5. 5Check active job status and pod state: kubectl get pods -n -l job-name=
  6. 6Check logs of the running or failed job: kubectl logs job/ -n
  7. 7If the job has no logs (too old or already cleaned up), increase failedJobsHistoryLimit on the CronJob to retain more history
  8. 8If events show 'too many missed start times', the controller has stopped scheduling. Delete and recreate the CronJob to reset the counter.
  9. 9Verify the schedule expression and timezone: kubectl get cronjob -n -o jsonpath='{.spec.schedule} {.spec.timeZone}'

Useful commands

kubectl get cronjob <name> -n <namespace>kubectl describe cronjob <name> -n <namespace>kubectl get jobs -n <namespace> -l job-name=<name>kubectl get pods -n <namespace> -l job-name=<name>kubectl logs job/<job-name> -n <namespace>kubectl get events -n <namespace> --field-selector involvedObject.name=<name>kubectl create job --from=cronjob/<name> <manual-job-name> -n <namespace>

Possible Resolutions

  • CronJob had no `activeDeadlineSeconds`. A job hung on a network call ran for hours, blocking the next scheduled run (concurrencyPolicy: Allow). Setting `activeDeadlineSeconds: 3600` and `concurrencyPolicy: Forbid` resolved it.

  • More than 100 schedules were missed due to a cluster outage. The controller silently stopped scheduling. Deleting and recreating the CronJob reset the missed-schedule counter and schedules resumed.

Dependencies

  • kube-state-metrics (for CronJob and Job metrics)
  • External dependencies the job calls (APIs, databases, storage)

Escalation

Guidance

Escalate if the stuck CronJob performs critical operations (backups, billing, data sync) and manual job creation has not helped, or if the job is consuming significant cluster resources.

Slack channels

  • #platform-oncall — Fill in: channel URL

Definitions

Alert rule

Fill in: link to the alert rule in _data/rules.yml on GitHub

Tuning notes

Set `activeDeadlineSeconds` on every CronJob to enforce a maximum run time. Use `concurrencyPolicy: Forbid` for jobs that must not overlap. Set `successfulJobsHistoryLimit: 3` and `failedJobsHistoryLimit: 5` to retain enough history without unbounded growth.

Related Links

Template adapted from https://runbooks.gitlab.com/template-alert-playbook/