Error Handling in SDKs, OTLP and the Collector: OTCA practice questions

OpenTelemetry treats telemetry as less important than the application that produces it. The error handling specification states the assumption plainly: users would rather lose telemetry than have the library change how their application behaves. Everything else in this topic follows from that choice.

For the API and SDK this means no unhandled exceptions at runtime. API methods must not throw when called incorrectly and should fall back to safe defaults, and the SDK must not throw for failures in its own work, such as an exporter that cannot reach its endpoint. Failing fast is allowed only during initialization, for example on a bad configuration, never later while the application runs. When the library suppresses an error it should log it, and the SDK internal logger level is controlled by OTEL_LOG_LEVEL, which defaults to info. SDKs must also let users register their own error handler, for instance to be strict in a staging environment.

Between an exporter and a receiver, OTLP defines which failures deserve a retry. Over OTLP/HTTP, only 429, 502, 503 and 504 should be retried, and every other 4xx or 5xx must not be; a 400 Bad Request signals permanently bad data. When the server sends a Retry-After header the client should honour it, otherwise it backs off exponentially. Over OTLP/gRPC, codes such as UNAVAILABLE, CANCELLED, DEADLINE_EXCEEDED, ABORTED, OUT_OF_RANGE and DATA_LOSS are retryable, while INVALID_ARGUMENT, PERMISSION_DENIED and UNAUTHENTICATED are not. RESOURCE_EXHAUSTED is retryable only when the server includes RetryInfo. For non-retryable errors the client must drop the data and should count what it dropped.

A successful response can still carry a partial_success field, with a count such as rejected_spans and a human-readable error_message. The client must not retry such a request, since the rest of the data was accepted.

The SDK OTLP exporter has an overall timeout, set through OTEL_EXPORTER_OTLP_TIMEOUT and defaulting to 10 seconds, that covers all attempts and waits for one batch and is not reset per attempt. Transient errors must be retried with exponential backoff and jitter within that budget.

In the Collector, most exporters share exporterhelper settings. The timeout option, 5 seconds by default, applies to each individual send attempt. retry_on_failure backs off from 5 seconds up to 30 seconds with a multiplier of 1.5 and stops after max_elapsed_time. The sending_queue, enabled by default with 10 consumers and room for 1000 requests, buffers data; with block_on_overflow left false, data that does not fit is rejected and shows up in enqueue-failed metrics. Pointing sending_queue.storage at a storage extension such as file_storage moves the queue to disk so it survives restarts.

The memory_limiter processor protects the Collector from running out of memory. Above its soft limit it refuses data with a non-permanent error so that the component in front of it can retry, and above the hard limit it also forces garbage collection. It should be the first processor in a pipeline so that this backpressure reaches the receivers.