Skip to content

Scaling the Collector: OTCA practice questions

Scaling starts with understanding the workload. Each signal can have different needs: scraping hundreds of Prometheus endpoints, ingesting a flood of logs and receiving OTLP traces are separate problems, and the scaling guide suggests giving them separate groups of Collectors so failures stay isolated and each group can grow on its own.

By Ibrahim Yesilay · Published · Updated

Independent community practice, not affiliated with or endorsed by CNCF or The Linux Foundation.

What Scaling the Collector covers in OTCA

Scaling starts with understanding the workload. Each signal can have different needs: scraping hundreds of Prometheus endpoints, ingesting a flood of logs and receiving OTLP traces are separate problems, and the scaling guide suggests giving them separate groups of Collectors so failures stay isolated and each group can grow on its own.

The Collector's own metrics tell you when to act. If the memory_limiter processor is in the pipeline, otelcol_processor_refused_spans and its equivalents for other signals count data that was blocked for memory reasons; frequent refusals suggest scaling up. The exporter queue metrics otelcol_exporter_queue_size and otelcol_exporter_queue_capacity show whether exporting keeps up with receiving, and the guide suggests scaling up around 60 to 70 percent of capacity while keeping a minimum number of replicas, such as three, for resilience. For scraping receivers, the trigger is a scrape cycle that takes almost as long as the scrape interval.

The same metrics also show when scaling will not help. If the queue keeps rising in step with every capacity increase, or otelcol_exporter_send_failed_spans keeps climbing, the bottleneck is probably the backend or the network, and more Collectors or more workers only add pressure.

For scaling purposes, components fall into three groups. Most are stateless, so you add replicas behind a load balancer. With gRPC, which OTLP commonly uses, that balancer should understand gRPC at layer 7, because a layer 4 balancer can pin each client connection to a single Collector. In Kubernetes, DaemonSets put a Collector on each node, while sidecars can balance gRPC better when there are few nodes and many pods, and they avoid losing every pod's pipeline when one node agent fails.

Scrapers such as the Prometheus receiver cannot be copied blindly, since identical replicas scrape the same targets and produce duplicates and out-of-order samples. You either give each Collector its own slice of targets or use the Target Allocator from the OpenTelemetry Operator. When it is enabled, the Operator replaces the static discovery in each scrape job with http_sd_configs pointing at the allocator, which spreads targets evenly across the Collector pods.

Stateful components such as the tail sampling processor, or span-derived service metrics, need every related item on one instance. The answer is a front layer of Collectors running the load-balancing exporter, which hashes the trace ID or the service name to choose a backend Collector. With a DNS resolver against a headless service, lower the resolution interval in elastic clusters so the tier views stay consistent. For managed scaling, the Operator can create a HorizontalPodAutoscaler from spec.autoscaler for deployment or statefulset modes, provided a Metrics Server is running.

Key points

  • Frequent memory_limiter refusals, reported as otelcol_processor_refused_spans, suggest scaling up.
  • Rising otelcol_exporter_send_failed_spans points at the backend, where adding Collectors can make things worse.
  • Identical Prometheus scrapers duplicate work; shard the targets or use the Target Allocator.
  • Stateful processing needs the load-balancing exporter to route by trace ID or service name.
  • Use a gRPC-aware layer 7 load balancer, because layer 4 balancing can pin clients to one Collector.
  • Operator-managed HPA works only for deployment and statefulset modes and needs a Metrics Server.

Three sample questions

17 OTCA questions on Scaling the Collector, 17 with explanations. How practice works

Question 1

What action should you take if the 'otelcol_exporter_queue_size' frequently approaches or exceeds the 'otelcol_exporter_queue_capacity'?
  1. A.Reduce the memory limit allocated to the Collector process to free RAM.
  2. B.Raise the queue capacity or, if the backend has room, add workers
  3. C.Decrease the number of exporter workers.
  4. D.Ignore the metrics as they are not critical.

Correct answer: B

The scaling page says that when otelcol_exporter_queue_size keeps getting close to otelcol_exporter_queue_capacity, exporting is slower than receiving: data piles up when there are not enough workers or the backend is too slow. The key follows the page: it suggests increasing the queue capacity (at the cost of memory) and notes that more workers often export more data. It also warns that more workers will not help when the backend itself is saturated. The internal telemetry page lists both metrics as gauges for this check. Why the other options are wrong. "Reduce the memory limit allocated to the Collector…": less memory makes the queue fill and data drop sooner. "Decrease the number of exporter workers.": fewer workers drain the queue more slowly and make the backlog worse. "Ignore the metrics as they are not critical.": the docs single these metrics out as signs of trouble; once the queue is full, data is rejected.

Sources: OpenTelemetry docs: Scaling the Collector, OpenTelemetry docs: Collector internal telemetry

Question 2

The memory_limiter processor is in every pipeline and otelcol_processor_refused_spans keeps increasing across the Collector cluster. What do the scaling docs suggest?
  1. A.Remove the memory_limiter, since refusals mean it is misconfigured
  2. B.Add more exporter workers, which reduces memory use in the pipeline
  3. C.Lower the scrape interval of the Prometheus receiver to reduce load
  4. D.Scale up the Collector cluster, since data is refused too often

Correct answer: D

The scaling page says that when memory_limiter is in the pipeline, otelcol_processor_refused_spans records data blocked by the processor, and the same metric exists for other signals. If data is being refused from entering the pipeline too often, you will probably want to scale up the Collector cluster, and you can scale down once memory use is clearly below the limit. Why the other options are wrong. "Remove the memory_limiter, since refusals mean it…": refusals are the processor working as designed; removing it risks out-of-memory crashes. "Add more exporter workers, which reduces memory us…": the docs warn that adding workers mainly pushes more data out and can hurt an overloaded backend; it is not a memory fix. "Lower the scrape interval of the Prometheus receiv…": the refused-spans metric points to Collector capacity, not to Prometheus scrape settings.

Sources: OpenTelemetry docs: Scaling the Collector, Collector core README: Memory limiter processor

Question 3

A Collector's otelcol_exporter_send_failed_spans keeps rising, meaning data fails permanently when sent to the backend. According to the scaling docs, what is the effect of adding more Collector replicas?
  1. A.It will likely worsen things, since the backend is the problem
  2. B.It fixes the failures because each replica gets a new retry budget
  3. C.It has no effect at all, since send failures are only metrics
  4. D.It is required, because send failures mean the queue has no workers

Correct answer: A

Under "When NOT to scale", the scaling page says an increase in otelcol_exporter_send_failed_spans indicates that sending to the backend failed permanently, and that scaling up the Collector will likely only worsen the situation when this keeps happening. Why the other options are wrong. "It fixes the failures because each replica gets a…": more replicas send more data to a backend that is already failing; retries do not fix the root cause. "It has no effect at all, since send failures are o…": the metric reflects real data loss, and scaling has a real (negative) effect. "It is required, because send failures mean the que…": the docs tie this metric to backend problems, not to a lack of workers.

Sources: OpenTelemetry docs: Scaling the Collector, Collector core README: Exporter helper (queue, batch, retry)

Practice this topic

Focus mode keeps every question on Scaling the Collector until you leave it. 10 free guest answers, then a free account; no credit card.

Or open OTCA practice across all topics.

Sources

The explanation above was checked against these official documents.

Frequently asked questions

Should I route by traceID or by service when generating span metrics across several Collectors?

By service. The gateway docs say service routing sends all spans of a service to the same Collector, which keeps per-service aggregations accurate.