Alerting & Dashboarding: PCA practice questions

Alerting in Prometheus is split between two programs, and most questions on this topic test whether you know which one does what. The Prometheus server evaluates alerting rules. Alertmanager turns the resulting alerts into notifications. Prometheus never sends an email or a page itself: it periodically sends the state of its alerts to one or more Alertmanager instances, which it can find through service discovery, and Alertmanager takes care of summarization, rate limiting, silencing and dispatching.

Alerting rules live in rule files, the same files that hold recording rules, and they are loaded by the server. Each rule has a name, a PromQL expression, and optional for, labels and annotations fields. Every series the expression returns becomes an active alert. Without a for clause the alert fires on the first evaluation that returns it. With for: 10m the alert first sits in the pending state and only moves to firing once the condition has held for ten minutes, which keeps short spikes from paging anyone. The keep_firing_for field does the reverse: it keeps an alert firing for a while after the condition clears, to ride out flapping and gaps in data. If no rules are defined, the server simply produces no alerts. Prometheus also exposes a synthetic ALERTS series with the value 1 for every pending or firing alert, which you can query like any other metric.

Labels and annotations look similar but serve different purposes. Labels are attached to the alert itself, overriding any conflicting label of the series, so they become part of what identifies it. Annotations carry longer, human-readable text such as a summary, a description or a link to a runbook or console, and both fields can be templated with $labels and $value.

Alertmanager has a small vocabulary worth knowing exactly. Grouping collects alerts of a similar nature into a single notification, so one outage that trips the same alert on many instances can arrive as one message. Inhibition suppresses notifications for some alerts while certain other alerts are already firing. A silence mutes matching alerts for a fixed period, for example during planned maintenance. Notifications go to receivers chosen by a routing tree in the configuration file, and several Alertmanagers can be clustered for high availability.

Recording rules belong here too because dashboards depend on them. A recording rule precomputes an expression that is frequently needed or expensive and stores the result as a new time series, so a dashboard that refreshes every few seconds reads one cheap series instead of recomputing a heavy aggregation. Rules in a group run sequentially at the group interval, which defaults to the global evaluation_interval.

The Prometheus documentation's advice on alert design is short: keep alerting simple, alert on symptoms that are associated with user pain rather than on every possible cause, avoid pages where there is nothing to do, and make alerts link to the consoles that help find the faulty component. The built-in expression browser at /graph runs any query as a table or a graph, but the documentation describes it as primarily useful for ad-hoc queries and debugging and points to Grafana or console templates for real dashboards.