Instrumentation and Exporters: PCA practice questions

Prometheus can only store what a target exposes, so this topic is about how metrics get onto a /metrics endpoint in the first place. There are two routes. Direct instrumentation means adding a Prometheus client library to your own code and defining metrics next to the code they measure. Exporters cover the cases where that is not feasible: they are separate programs that translate the existing statistics of a third-party system, such as Linux kernel counters or an HAProxy status page, into Prometheus metrics. The node exporter is the official exporter for hardware and operating system metrics on a host. The blackbox exporter takes a different angle: it probes endpoints from the outside over HTTP, HTTPS, DNS, TCP, ICMP and gRPC, so you can measure availability and latency without touching the target.

Every scraped series also carries two labels that Prometheus attaches itself. The job label is the configured job name the target belongs to, a job being a collection of instances with the same purpose. The instance label is the host:port part of the URL that was scraped. For each scrape Prometheus also records synthetic series such as up, which is 1 when the target was reachable and 0 when the scrape failed, and scrape_duration_seconds.

The instrumentation guidelines group services into three kinds. Online-serving systems should expose the number of requests, errors and latency, ideally on both the client and the server side. Offline processing pipelines should count items coming in, items in progress and items sent out, plus the time something was last processed. Batch jobs care most about the time of the last successful run. For a batch job the guidelines suggest recording that time, the duration of each stage and the total runtime as gauges and pushing them to the Pushgateway, because the job may be gone before Prometheus could scrape it.

Choosing the metric type is part of instrumentation. Use a counter for something that only goes up, such as requests served, and a gauge for anything that can fall, such as memory in use or requests in progress. For latency, a histogram records observations in configurable buckets plus _sum (the total of all observed values) and _count (the number of observations); because buckets from many instances can be summed before computing a quantile, histograms are the type to choose when you want service-wide percentiles later.

Naming and labels follow written conventions. A metric name should start with a single-word prefix for its domain, use base units such as seconds and bytes rather than milliseconds or megabytes, end with the unit in plural form, and end with _total when it is an accumulating counter. myapp_requests_total with a code label is the shape the guidelines recommend: one metric with a label instead of a separate metric per status code. Labels must not hold unbounded values such as user IDs or email addresses, because every distinct combination of label values is a new time series with its own memory and storage cost. The guidelines suggest keeping cardinality low and also recommend exporting known series with a default of 0 so they exist before the first event happens.