Debugging Observability Pipelines: OTCA practice questions

Keeping a telemetry pipeline healthy means being able to tell where data stops. The Collector documentation suggests walking the path hop by hop and asking the same questions at each one: what the logs say, how data is received and modified, how it is exported, in which format, how the next hop is configured, and whether network policies block traffic in or out.

The Collector reports on itself. Its internal metrics are exposed in Prometheus format on port 8888 by default, at a verbosity of none, basic, normal or detailed, with normal as the default, and its logs go to stderr at INFO unless the level is changed under service.telemetry. A few metrics answer most questions. otelcol_receiver_accepted_spans (and the equivalents for metric points and log records) counts what entered a pipeline, while otelcol_receiver_refused_spans counts what could not be pushed in. otelcol_exporter_sent_spans counts successful sends and otelcol_exporter_send_failed_spans counts failed attempts, which do not mean data loss by themselves because a retry may succeed. otelcol_exporter_enqueue_failed_spans counts data that never reached the sending queue, and otelcol_exporter_queue_size against otelcol_exporter_queue_capacity shows how full the queue is. A persistent gap between accepted and sent data means data is being lost or held back.

For direct inspection, the debug exporter prints what passes through, and verbosity: detailed prints whole payloads; sending a test payload through it confirms that receiving, processing and exporting work. The zpages extension, on port 55679 locally, shows live data from receivers and exporters, and its TraceZ page helps with latency and stuck spans. The pprof extension on port 1777 profiles a running Collector, and otelcol components lists the components in a distribution with their stability.

Typical symptoms have typical causes. No data received points at network settings, the client and receiver configuration, or a receiver that is configured but not listed in any pipeline. Data that is not exported points at firewalls, DNS or proxies, the exporter configuration, or an unavailable destination. Dropped data usually means an undersized Collector or a destination that is down or slow. A Collector that exits or restarts is often under memory pressure, for example from a missing memory_limiter processor, a queue that is too large, or container limits.

Resilience is configured on exporters. The sending queue buffers data in memory, retry_on_failure retries with exponential backoff for up to five minutes by default, and once the queue is full new data is dropped to protect memory. Pointing the queue at the file_storage extension turns it into a write-ahead log on disk that survives restarts. Data is still lost if the destination is down longer than the retry window, the queue fills, or the Collector crashes with only an in-memory queue.

Scaling decisions follow from the same signals. Refusals from the memory_limiter, visible as otelcol_processor_refused_spans, and a queue staying above roughly 60 to 70 percent of capacity suggest more instances; a rising send failure count suggests the backend, where more Collectors would only make things worse. On the SDK side, metric views can drop attributes that are not needed before data ever leaves the application.