Analysis and Outcomes: From Telemetry to Answers: OTCA practice questions

Collecting telemetry is only useful if it helps people answer questions about a running system. OpenTelemetry's observability primer frames the goal as understanding a system from the outside: you can ask new questions without knowing its inner workings, troubleshoot novel problems, often called unknown unknowns, and answer why something is happening. A service is properly instrumented when engineers no longer need to add instrumentation to debug an issue, because the data they need is already there.

Reliability is judged from the user's side. The primer's example is a shop that stays up all the time but sometimes puts the wrong shoes in the cart, which makes it unreliable despite perfect uptime. A Service Level Indicator, or SLI, measures how a service behaves, and good SLIs take the user's perspective, such as how fast a page loads. A Service Level Objective, or SLO, attaches one or more SLIs to business value so reliability can be communicated across an organization.

Much of the value comes from correlating signals. The logs specification names three correlation dimensions: time of execution, execution context through trace and span IDs on log records, and origin through the resource attached to each record. SDKs and automatic instrumentation can stamp the active trace context onto log records, so a log line leads straight to its trace. Exemplars do the same for metrics. An exemplar is a sample measurement that can carry the trace ID and span ID that were active when it was recorded, along with its time, value and filtered attributes. With the default TraceBased filter, measurements recorded inside a sampled span become exemplar candidates, so a latency spike on a histogram can lead to a concrete trace. Profiles extend this further by linking slow spans or resource spikes to the code that consumed the resources.

Analysis also has costs. Metric cardinality is the number of unique attribute combinations, and the SDK keeps one aggregation state for each. To bound memory, the metrics SDK caps combinations per stream, 2000 by default. Beyond that limit, new combinations are folded into one overflow point tagged otel.metric.overflow=true. Totals stay correct, but queries that filter on any attribute undercount, because the whole combination is replaced. That can silently break an error-rate alert, so keep high-cardinality values such as user IDs or raw paths off metric attributes.

Sampling is the other trade-off. Head sampling decides early, for example by keeping a fixed share of trace IDs, and it is simple and efficient. However, it cannot guarantee that every trace containing an error is kept. Tail sampling decides after seeing most of a trace, so it can keep error or slow traces, but it needs stateful infrastructure to run. Sampling may not be worth it when traffic is tiny, when data is only used in aggregate, or when regulation forbids dropping data.