# Reduce Telemetry Ingestion Costs

Telemetry data volume grows fast. Verbose debug logs, health-check traces, and redundant metric labels can quietly inflate your observability bill without adding signal. Bindplane gives you pipeline-level controls to trim that waste before data ever reaches a paid destination — without touching your instrumentation.

This guide covers four practical strategies:

1. [Filtering](#filtering-drop-what-you-dont-need) — drop logs, metrics, and traces that have no operational value
2. [Field redaction](#field-redaction-shrink-payload-size) — strip sensitive or redundant fields to reduce payload size
3. [Sampling](#sampling-send-a-representative-fraction) — send only a statistical fraction of high-volume streams
4. [Routing](#routing-cheaper-destinations-for-low-priority-data) — send low-priority data to cheaper storage tiers

A fifth section covers [identifying your highest-volume sources](#identify-your-highest-volume-sources) so you know where to focus first.

{% hint style="success" %}
Combining filtering, field redaction, and sampling together is how Bindplane customers achieve up to 40% reduction in ingestion costs without losing the data that matters.
{% endhint %}

***

### Filtering — Drop What You Don't Need

Filtering is the highest-leverage strategy: data that is never sent costs nothing. Common candidates include:

* `DEBUG` and `TRACE` logs in production
* Kubernetes liveness/readiness health-check HTTP logs
* Metrics emitted by unused infrastructure components
* Traces for synthetic monitoring traffic

#### Filter by severity

The [Filter Severity processor](/integrations/processors/filter-severity.md) drops all logs below a minimum severity level. To keep only `WARN`, `ERROR`, and `FATAL` logs:

1. Open your configuration in Bindplane.
2. Click the processor node between your source and destination.
3. Add the **Filter Severity** processor and set **Minimum Severity** to `WARN`.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: drop-debug-logs
spec:
  type: filter_severity
  parameters:
    - name: severity
      value: WARN
```

After rollout, the throughput counter on the destination node will drop immediately, reflecting fewer events being forwarded.

See the [Reduce Log Volume with the Severity Filter](/how-to-guides/data-collection-and-processing/reduce-log-volume-with-the-severity-filter.md) how-to guide for a full walkthrough with screenshots.

#### Filter by condition

The [Filter by Condition processor](/integrations/processors/filter-by-condition.md) accepts any OTTL expression, giving you precise control over which records are dropped. This is the recommended approach for most filtering use cases.

**Example: drop health-check HTTP logs**

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: drop-health-checks
spec:
  type: filter-by-condition
  parameters:
    - name: telemetry_types
      value:
        - Logs
    - name: action
      value: exclude
    - name: condition
      value:
        ottl: (attributes["http.target"] == "/healthz" or attributes["http.target"] == "/readyz")
        ui:
          statements:
            - key: http.target
              match: attributes
              operator: Equals
              value: /healthz
            - key: http.target
              match: attributes
              operator: Equals
              value: /readyz
```

**Example: drop metrics by name**

Use the [Filter Metric Name processor](/integrations/processors/filter-metric-name.md) to exclude entire metric families you are not actively using:

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: drop-unused-k8s-metrics
spec:
  type: filter_metric_name
  parameters:
    - name: action
      value: exclude
    - name: match_type
      value: regexp
    - name: metric_names
      value:
        - k8s.node.condition.*
        - k8s.pod.network.*
```

{% hint style="info" %}
Use [Snapshots](/feature-guides/data-management/snapshots.md) to inspect live telemetry and identify which fields and values to filter on before committing a processor to production.
{% endhint %}

***

### Field Redaction — Shrink Payload Size

Even when you want to keep a log record, individual fields within it may be redundant or sensitive. Removing those fields reduces the byte size of every event that reaches your destination.

#### Remove specific fields

The [Delete Fields processor](/integrations/processors/delete-fields.md) removes named attributes, resource attributes, or log body keys:

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: trim-verbose-fields
spec:
  type: delete_fields-v2
  parameters:
    - name: telemetry_types
      value:
        - Logs
    - name: attributes
      value:
        - log.file.name
        - process.pid
        - thread.id
    - name: resource_attributes
      value:
        - host.id
        - os.description
```

High-cardinality fields like request IDs, session tokens, and container UIDs can add significant per-event overhead without being useful for alerting or dashboards. Dropping them at the pipeline level can reduce average log size by 20–40%.

#### Redact sensitive data

The [Redact Sensitive Data processor](/integrations/processors/redact-sensitive-data.md) detects and replaces sensitive values — SSNs, credit card numbers, email addresses, IP addresses, and more — using built-in rule presets or custom regex patterns. Aside from compliance benefits, redacting large PII strings trims payload size.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: redact-pii
spec:
  type: redact_sensitive_data
  parameters:
    - name: telemetry_types
      value:
        - Logs
    - name: redaction_strategy
      value: Censor with Asterisks
    - name: redaction_rules
      value:
        - Email
        - IPv4 Address
        - Social Security Number (SSN)
        - Credit Card
```

{% hint style="success" %}
The [Process Logs for ClickHouse Blueprint](https://app.bindplane.com/p/01J06XSD7F4KT3D0XDE2VQDAR5/blueprints) combines debug-log filtering, PII redaction, high-cardinality field removal, and deduplication into a single ready-to-use processor bundle. See the [Blueprints guide](/feature-guides/pipeline-processing/blueprints.md) for more on using pre-built processor bundles.
{% endhint %}

***

### Sampling — Send a Representative Fraction

For high-volume, repetitive streams — think HTTP access logs or application traces with thousands of requests per second — probabilistic sampling lets you forward a statistically representative subset while discarding the rest.

#### Log sampling

The [Log Sampling processor](/integrations/processors/log-sampling.md) drops a configurable percentage of matching log records. A `drop_ratio` of `0.75` forwards only 25% of matching entries:

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: sample-access-logs
spec:
  type: sampling
  parameters:
    - name: drop_ratio
      value: '0.75'
    - name: condition
      value: '(attributes["log_type"] == "http_access")'
```

{% hint style="info" %}
Sampling is most appropriate for high-frequency, low-severity informational logs. For error and warning logs, prefer filtering by condition so that every anomaly is preserved.
{% endhint %}

**Choosing a drop ratio**

| Stream type                              | Suggested drop ratio | Retained fraction |
| ---------------------------------------- | -------------------- | ----------------- |
| HTTP access logs (INFO)                  | `0.90`               | 10%               |
| Application trace spans (non-error)      | `0.75`               | 25%               |
| Debug logs not caught by severity filter | `0.95`               | 5%                |
| Warning logs                             | `0.00`               | 100% (keep all)   |
| Error / Fatal logs                       | `0.00`               | 100% (keep all)   |

{% hint style="warning" %}
Never sample `ERROR` or `FATAL` log records. Use the [Filter by Condition processor](/integrations/processors/filter-by-condition.md) with `action: include` on those severities to ensure a separate path keeps every high-severity event intact.
{% endhint %}

***

### Routing — Cheaper Destinations for Low-Priority Data

Not all telemetry has the same business value. You can use Bindplane's routing capabilities to send high-priority data to a full-featured observability platform while directing lower-priority telemetry to cheaper long-term storage — object storage, ClickHouse, or a cold-tier SIEM.

#### Route by severity or label

Use the [Filter by Condition processor](/integrations/processors/filter-by-condition.md) in front of each destination to control what flows where:

* Place a severity filter (minimum `WARN`) in front of your expensive destination to forward only actionable events.
* Place an inverse filter (maximum `INFO`) in front of your cheap cold-storage destination to capture verbose logs inexpensively.

The [Routing Telemetry](/how-to-guides/data-collection-and-processing/routing-telemetry.md) how-to guide provides a step-by-step example of excluding or including logs by attribute value for specific destinations.

#### Multi-destination pipelines

Bindplane supports multiple destinations per configuration. You can fan out telemetry and apply different processor stacks per destination branch:

```
Source → [Severity Filter ≥ WARN] → Premium SIEM
       → [Severity Filter ≤ INFO]  → Cold Storage (S3 / ClickHouse)
```

This lets you keep full fidelity in cheap storage for forensics while paying premium rates only for the events worth alerting on.

***

### Identify Your Highest-Volume Sources

Before applying any reduction strategy, profile which sources and log types are generating the most data. Bindplane shows throughput metrics per pipeline node in the configuration view.

1. Navigate to **Configs** and open any active configuration.
2. The throughput counter on each source node shows events per second and approximate bytes per second.
3. Click into a collector attached to the configuration and select **View Recent Telemetry** to inspect live events using [Snapshots](/feature-guides/data-management/snapshots.md).
4. Sort by volume and focus reduction efforts on the top two or three sources first.

{% hint style="success" %}
Start with the highest-volume, lowest-severity stream. Applying a severity filter to a chatty debug-heavy service is often enough to achieve a 30–40% volume reduction on its own.
{% endhint %}

#### Using Blueprints for a head start

[Blueprints](/feature-guides/pipeline-processing/blueprints.md) are pre-built processor bundles available in the Bindplane library. Several Blueprints are purpose-built for volume reduction, including one that combines JSON parsing, PII redaction, high-cardinality field removal, deduplication, and debug-log filtering into a single click.

To apply a Blueprint:

1. Navigate to **Blueprints** in the Bindplane UI.
2. Find a relevant Blueprint (for example, **Process Logs for ClickHouse** for a combined log hygiene pipeline).
3. Click **Add to Project** and provide a name.
4. Add the resulting processor bundle to your pipeline configuration.

***

### Summary

| Strategy                            | Processor(s)                                                               | Typical volume reduction |
| ----------------------------------- | -------------------------------------------------------------------------- | ------------------------ |
| Filter debug/trace logs             | [Filter Severity](/integrations/processors/filter-severity.md)             | 20–50%                   |
| Filter noisy events by field        | [Filter by Condition](/integrations/processors/filter-by-condition.md)     | 10–30%                   |
| Drop unused metrics                 | [Filter Metric Name](/integrations/processors/filter-metric-name.md)       | 5–20%                    |
| Remove verbose fields               | [Delete Fields](/integrations/processors/delete-fields.md)                 | 10–30% per event         |
| Redact and shrink PII fields        | [Redact Sensitive Data](/integrations/processors/redact-sensitive-data.md) | 5–15% per event          |
| Probabilistic log sampling          | [Log Sampling](/integrations/processors/log-sampling.md)                   | Up to 95%                |
| Route low-priority to cheap storage | Filter by Condition + multi-destination                                    | Variable                 |

Combining two or three of these strategies on your highest-volume sources is typically enough to reach a 40% or greater reduction in total ingestion cost.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.bindplane.com/how-to-guides/data-collection-and-processing/reduce-telemetry-ingestion-costs.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.