Transforming Telemetry in the Collector: OTCA practice questions

The Collector is a convenient place to reshape telemetry before it leaves your infrastructure, usually for data quality, governance, cost or security. Contrib processors offer dozens of transformations, and the documentation warns that heavy transformations can noticeably affect Collector performance.

Much of this work now runs on the OpenTelemetry Transformation Language, OTTL. An OTTL statement combines a function, such as set or delete_key, with an optional where condition that decides whether the function runs. Paths reach into the data through contexts like resource, scope, span, spanevent, metric, datapoint, exemplar and log. OTTL cannot cross signals, so a log statement cannot read a span field.

The transform processor executes OTTL statements in order, with separate lists named trace_statements, metric_statements and log_statements. Each list accepts only the contexts that belong to its signal. Its error_mode defaults to ignore, which logs the error and continues. The silent mode skips logging, and propagate returns the error up the pipeline, dropping the payload. The older attributes processor, which inserts, updates, upserts, deletes, hashes, extracts or converts attributes, and the resource processor, which does the same for resource attributes, both now state that they are superseded by the transform processor. For metric renames, the metrics transform processor remains an option.

Dropping data is the filter processor's job, and it also uses OTTL. Telemetry that matches any configured condition is dropped, because the conditions are ORed. Conditions are evaluated from higher levels to lower ones. Removing every span event leaves the span in place, but removing every data point of a metric removes the metric. For sensitive values, the redaction processor keeps only allowed attribute keys and can mask blocked values, and its list of allowed keys fails closed when left empty.

Enrichment usually comes from two processors. The resource detection processor adds host or cloud information from detectors such as env, system, ec2, gcp or azure, and when two detectors set the same attribute the first one listed wins. The Kubernetes attributes processor adds pod, namespace, node and workload metadata and needs RBAC read access to the Kubernetes API.

Sampling is a transformation too. The probabilistic sampler processor makes stateless decisions from the trace ID and is the efficient choice when you need a fixed percentage. The tail sampling processor holds spans in memory, for 30 seconds by default, and then applies policies such as latency, status code or OTTL conditions to the whole trace, which requires all spans of a trace to reach one instance. Batching closes the loop. Exporters can batch inside their sending queue, which is off until you configure a batch block, and the separate batch processor belongs after memory_limiter and any sampling.