Prometheus Fundamentals: PCA practice questions

Prometheus is a monitoring system built around a time series database. The main server scrapes metrics from instrumented targets over HTTP, stores every sample locally, evaluates rules and answers PromQL queries. Around it sit client libraries for instrumenting code, the Pushgateway for short-lived jobs, exporters for third-party systems and Alertmanager for notifications. Each server works on its own and does not depend on distributed storage, a design choice that explains several limitations you need to know.

The data model is the foundation. Every time series is uniquely identified by its metric name plus an optional set of key-value labels, and changing, adding or removing any label value produces a different series. A sample is a float64 value, or a native histogram, with a millisecond-precision timestamp. Internally the metric name is just the label __name__, so http_requests_total and {__name__="http_requests_total"} select the same thing, and every label name that starts with two underscores is reserved for internal use. The number of distinct series a metric produces is its cardinality; labels with unbounded values such as user IDs multiply it and drive memory and storage use up.

On the wire, a scraped /metrics endpoint returns the text exposition format: each metric name can have one # HELP line with a docstring and a # TYPE line, placed before its first sample, declaring counter, gauge, histogram, summary or untyped; a metric without a TYPE line is untyped. A histogram exposes cumulative _bucket series labelled with le (the upper bound), including a required le="+Inf" bucket that equals _count, plus _sum and _count. A summary exposes precomputed quantile series plus _sum and _count.

Configuration lives in a YAML file passed with --config.file. The global section defaults to a scrape_interval of 1m, a scrape_timeout of 10s and an evaluation_interval of 1m, and each scrape job defaults to the /metrics path over http. A running server reloads its configuration and rule files on SIGHUP, or on an HTTP POST to /-/reload when it was started with --web.enable-lifecycle; a malformed new configuration is not applied. For Kubernetes, kubernetes_sd_configs discovers targets with one of the roles node, service, pod, endpoints, endpointslice or ingress. The pod role turns every declared container port of every pod into a target and exposes pod labels as __meta_kubernetes_pod_label_<name> meta labels, which relabeling rules can match on to decide which pods are scraped.

Storage is a local on-disk time series database. Samples are grouped into two-hour blocks that are compacted later, recent data is protected by a write-ahead log, and data is kept for 15 days unless you set another retention time or size. Local storage is not clustered or replicated, so it should be managed like a single-node database. Long-term or durable storage is reached through the remote write and remote read interfaces to external systems. High availability is achieved by running identical Prometheus servers side by side; Alertmanager deduplicates the identical alerts they both send, and Alertmanager itself can be clustered.