Reading Prometheus Alert Rules: A Practical Reference for Debugging Firing Alerts

Prometheus alert rules are the backbone of any monitoring setup. I've seen teams copy rules from example repositories without understanding a single line of PromQL, then wonder why they get paged for garbage. Reading alert rules is a skill—you need to parse the expression, interpret the intent behind the thresholds, and spot the subtle bugs that can cause false positives or, worse, missed alerts.

This isn't a tutorial on writing alert rules. This is about reading them. You're on-call, an alert fires, and you need to understand what the rule author was thinking. Or you're reviewing a PR that adds a rule and you want to catch mistakes before they hit production. Let's get into the actual mechanics.

Every rule has four components. The expr is the PromQL expression that evaluates to a vector of instant vectors; each element becomes a potential alert. The for clause adds a duration window. Labels are attached to the alert and used for routing and deduplication in Alertmanager. Annotations provide human-readable context—these are what you see in the notification.

The most common mistake I see is overloading labels with information that should be in annotations. Labels are for identity—they determine how alerts group and route. Annotations are for the human. If you put the current value in a label, every value change creates a new alert fingerprint, and you'll get duplicates. Keep labels stable.

The expression is where the logic lives, and it's where reading gets hard. Let's pick apart the example above. The expression computes the ratio of 5xx requests to total requests over the last 5 minutes. The denominator uses a rate on all status codes; the numerator only on 5xx. If the total rate is zero (no traffic), the division yields NaN, and the condition '> 0.05' is false—so no alert. That's correct behavior, but I've seen rules that forget to handle division by zero and cause spurious alerts.

Another trap: using 'increase()' instead of 'rate()' for thresholding. 'increase()' gives a raw count over the time window, which is sensitive to the window length. A 5-minute increase of 50 errors might be fine at low traffic but critical at high traffic. 'rate()' normalizes per second, making thresholds portable across different load levels.

Also watch out for range vectors. The [5m] in the example is the lookback window for the rate. If your scrape interval is 15s, 5m gives you 20 data points—enough for a stable rate. But if you use [1m] with a 15s scrape, you only get 4 points, and the rate can jump around. Always check that the range vector is at least 4x the scrape interval.

The 'for' clause is the minimum duration the condition must be continuously true before the alert fires. This prevents flapping. However, the evaluation happens at the rule evaluation interval (default 1m). If 'for: 5m', the condition must be true at each evaluation step for 5 consecutive minutes. If the metric is scraped every 30s, but evaluated every 1m, a 30-second spike might be missed entirely.

I once inherited a rule with 'for: 10m' on a latency alert. The SLO was 99.9%, but the team kept missing brief latency spikes that lasted 2–3 minutes. They'd resolve before the alert ever fired. The fix was to lower 'for' to 1m or use a recording rule that aggregates more granularly. The 'for' clause is not a band-aid for noisy metrics—fix the metric stability instead.

Annotations are what show up in PagerDuty, Slack, or email. A good annotation tells you what's broken, how bad it is, and where to look. The example above uses Go template syntax: '{{ $labels.instance }}' and '{{ $value }}'. You can use any label from the alert, plus built-in variables like $value, $labels, and $externalURL.

Common annotation fields: summary (one-liner), description (detailed), runbook_url (link to playbook), and dashboard (link to a Grafana panel). I recommend always including the current value and a link to a relevant dashboard. The worst annotation I've seen: 'description: Something is wrong'. That's not helpful. Be specific.

Labels in the alert rule are merged with the metric labels. They're used by Alertmanager to route alerts to the right receiver (e.g., team, severity). The most important label is 'severity'—commonly 'critical', 'warning', 'info'. You might also have 'team' or 'service'.

A common antipattern is using dynamic labels like 'instance' in the rule's label section. That's redundant because the metric already has that label. Only add labels that are not already present, or override metric labels for routing purposes. Overriding a metric label is rare and should be done with caution—it changes the alert identity.

Go templates in annotations are powerful, but they can also break silently. If a template references a label that doesn't exist on the alert, the template rendering fails and the notification might show an error string. Always test your templates with actual data.

You can test templates using the 'amtool' command-line tool or by sending a test alert to Alertmanager's API. I've seen templates that use '{{ $labels.team }}' but the rule doesn't have a 'team' label—the alert fires but the notification is blank. That's frustrating.

(That stat is made up, but it feels true based on my experience.) Point is: test your templates.

Once you're comfortable reading standard rules, you'll encounter patterns like multi-dimensional alerts (using 'by' or 'without' in the expression), alerts based on recording rules, and alerts that use 'absent()' for dead man's switches. The same principles apply: understand the metric, the aggregation, and the intent.

One pattern I like is using a recording rule to pre-process a complex expression, then alert on the recording rule. This separates the computation from the alerting logic and makes both easier to read. For example:

recording: 'job:error_rate:5m' with expression 'rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])'

Then the alert rule simply does 'job:error_rate:5m > 0.05'. Clean and testable.

Reading Prometheus alert rules is a skill that improves with practice. Next time you're on-call and an alert fires, don't just acknowledge it—read the rule. Understand the expression, the for clause, the annotations. You might find a bug that's been hiding for months. And if you do, file a PR to fix it. Your future self (and your teammates) will thank you.