Observability Fundamentals: OTCA practice questions

Observability is the ability to understand a system from the outside, by asking questions about it without knowing its internals, which is what makes it useful for problems nobody anticipated. For an application to be observable it must emit telemetry, the data a system produces about its own behavior, and OpenTelemetry is a way to instrument code so it does.

OpenTelemetry is an open source framework and toolkit for generating, exporting and collecting telemetry. It is vendor- and tool-agnostic, and it is explicitly not an observability backend: storing and visualizing the data is left to tools such as Jaeger or Prometheus or to commercial products. Its parts are a specification and the OTLP protocol, semantic conventions, an API, language SDKs, instrumentation libraries, zero-code instrumentation, the Collector, and tooling for Kubernetes. It is a Cloud Native Computing Foundation project that came out of the merger of OpenTracing and OpenCensus, two earlier efforts at the same problem.

The documentation lists the signals that OpenTelemetry supports. Traces follow a request's path through an application. Metrics are measurements taken at runtime, aggregated over time, such as an error rate or a request rate. Logs are timestamped records of events. Baggage carries contextual key-value data between signals. Events, a specific kind of log, and profiles, which record code-level resource usage, are listed as still under development.

A trace is made of spans. Each span is one unit of work with a name, start and end times, attributes, events and a status; a root span covers the whole request and child spans show the steps inside it. The span context, holding the trace ID and span ID, is what links spans across services. A span kind describes its role: client and server for synchronous remote calls, producer and consumer for asynchronous work, and internal for work inside one process, which is the default. Status is Unset by default, Error when something failed, and Ok only when someone explicitly marks the operation as successful.

Logs become far more useful when they can be correlated. With an active SDK or automatic instrumentation, log records carry the current trace and span IDs, so a log line leads to the trace it belongs to. Structured logs, which follow a consistent schema with defined fields, are preferred in production because they are easier to parse and analyze than free text, and valid JSON alone does not make a log structured.

Context propagation is what joins work done in different processes into one trace. Context holds the identifiers, and propagation moves them between services, by default using the W3C TraceContext traceparent header. Baggage travels the same way, so it should never contain credentials or personal data. Semantic conventions give common names to operations and attributes across traces, metrics, logs, profiles and resources, so that telemetry means the same thing whichever library produced it.

Observability supports reliability work: a service level indicator measures behavior from the user's point of view, such as page load time, and a service level objective sets the target for it.