# Configure Collector Queue Size

The Bindplane OTel Collector (BDOT) buffers outgoing telemetry in a **sending queue** before delivering it to a destination. Knowing how the queue works — and how to size it — is the key to controlling memory consumption, preventing disk bloat, and avoiding silent data loss when a destination becomes unreachable.

### How the Sending Queue Works

Every destination in a Bindplane configuration exposes a sending queue that sits between the pipeline and the export call. When the destination endpoint is slow or temporarily unavailable, the queue absorbs the backlog so the rest of the pipeline can keep running.

```
Receiver → Processors → [Sending Queue] → Destination
```

The queue holds **batches** of telemetry, not individual spans or log records. One queue slot = one batch produced by the batch processor upstream.

Two queue types are available:

| Type                           | Storage    | Survives collector restart            | Best for                               |
| ------------------------------ | ---------- | ------------------------------------- | -------------------------------------- |
| **In-memory** (default)        | RAM        | No — data is lost on crash or restart | Low-latency, non-critical telemetry    |
| **Persistent** (Pebble / Bolt) | Local disk | Yes                                   | Critical telemetry, gateway collectors |

{% hint style="info" %}
The in-memory queue is enabled by default. To survive restarts you must explicitly configure a [persistent queue](/configuration/bindplane-otel-collector/persistent-queue.md) backend (Pebble is recommended for new deployments).
{% endhint %}

### Key Parameters

These parameters appear on every destination's **Advanced** settings panel.

| Parameter                     | Default | Description                                                                                                                                                            |
| ----------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sending_queue_enabled`       | `true`  | Enables the sending queue. Disable only when the destination is always reachable and you want zero buffering overhead.                                                 |
| `sending_queue_queue_size`    | `5000`  | Maximum number of batches held in the queue. When the queue is full, new batches are **dropped**.                                                                      |
| `sending_queue_num_consumers` | `10`    | Number of concurrent goroutines reading from the queue and sending to the destination. Higher values increase throughput at the cost of more CPU and open connections. |

### Estimating Memory Usage

For the **in-memory queue**, the total RAM consumed is approximately:

```
memory_used ≈ queue_size × avg_batch_size_bytes
```

**Example:** Default `queue_size` of `5000` with a typical 100 KB batch:

```
5000 × 100 KB = 500 MB (worst case — queue completely full)
```

In practice, the queue drains continuously, so average memory usage is a fraction of the worst case. However, plan for the worst case when sizing collector nodes.

**Reducing memory pressure:**

* Lower `queue_size` (e.g., `500`–`1000`) on memory-constrained hosts.
* Reduce batch size in the [batch processor](/integrations/processors/batch.md) (`send_batch_max_size`).
* Switch to a [persistent queue](/configuration/bindplane-otel-collector/persistent-queue.md) to move storage off the heap.

### Estimating Disk Usage (Persistent Queue)

For a **persistent queue**, the same formula applies but the cost is disk space, not RAM:

```
disk_used ≈ queue_size × avg_batch_size_bytes
```

The persistent queue backend (Pebble) compacts old entries automatically. Monitor actual disk growth and set `queue_size` so the queue never fills a partition.

{% hint style="warning" %}
A full disk stops the persistent queue from accepting new entries, which causes back-pressure and eventually dropped data — the same outcome as a full in-memory queue.
{% endhint %}

### What Happens When the Queue Is Full

When `queue_size` is reached and the destination is still unavailable:

1. The queue stops accepting new batches.
2. Back-pressure propagates upstream — receivers and processors stall.
3. If back-pressure cannot be absorbed, **telemetry is dropped and a warning is logged**:

   ```
   Dropping data because sending_queue is full
   ```

Increase `queue_size` to absorb longer outages, or investigate why the destination is not keeping up (network issues, throttling, insufficient `num_consumers`).

### Retry on Failure vs. Queue Size

The sending queue and [retry-on-failure](/configuration/bindplane-otel-collector/retry-on-failure.md) work together but serve different purposes:

| Setting              | Purpose                                                             |
| -------------------- | ------------------------------------------------------------------- |
| **Sending queue**    | Holds batches while the destination is unreachable                  |
| **Retry on failure** | Re-attempts individual failed export calls with exponential backoff |

Both are enabled by default. The queue provides the buffer; retry on failure governs how aggressively the collector retries each dequeue attempt before giving up.

### Recommended Defaults and When to Tune

| Scenario                                           | Recommended `queue_size`           | Notes                                          |
| -------------------------------------------------- | ---------------------------------- | ---------------------------------------------- |
| Memory-constrained edge agent (< 512 MB RAM)       | `250`–`500`                        | Pair with a small `send_batch_max_size`        |
| Standard agent (1–4 GB RAM)                        | `1000`–`2000`                      | Fits most single-host workloads                |
| Gateway collector (high throughput)                | `5000`–`10000`                     | Use a persistent queue to avoid RAM exhaustion |
| Critical telemetry (must not lose data on restart) | Any, with persistent queue enabled | Pebble is recommended                          |

Increase `num_consumers` (e.g., `20`–`50`) on gateway collectors that send to low-latency destinations to maximize export throughput.

### What YOU Should Do Next

1. Check `queue_size` against available RAM: `queue_size × avg_batch_size ≤ available_memory × 0.5`
2. Enable a [persistent queue](/configuration/bindplane-otel-collector/persistent-queue.md) if data must survive a collector restart
3. Monitor queue depth using [internal telemetry](/configuration/bindplane-otel-collector/internal-telemetry.md)
4. Watch collector logs for `Dropping data because sending_queue is full` warnings
5. For gateway deployments, prefer [Pebble](/configuration/bindplane-otel-collector/persistent-queue/pebble.md) over the default in-memory queue

### Related Documentation

* [Sending Queue](/configuration/bindplane-otel-collector/sending-queue.md)
* [Persistent Queue](/configuration/bindplane-otel-collector/persistent-queue.md)
* [Retry on Failure](/configuration/bindplane-otel-collector/retry-on-failure.md)
* [Batching Configuration Performance Impact](/how-to-guides/data-collection-and-processing/batching-configuration-performance-impact.md)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.bindplane.com/how-to-guides/data-collection-and-processing/configure-collector-queue-size.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.