> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/processors/deduplicate-logs.md).

# Deduplicate Logs

The Deduplicate Logs processor watches the log stream over a fixed interval and merges identical records into one. The emitted record carries a count attribute with the number of logs that were collapsed, plus first- and last-observed timestamps. It operates on logs only.

### Supported Telemetry

| Metrics | Logs | Traces |
| ------- | ---- | ------ |
|         | ✓    |        |

### Configuration

<figure><img src="/files/uCiIKuUkST5ULrsvS9NM" alt="Bindplane docs - Deduplicate Logs - image 1"><figcaption></figcaption></figure>

**Matching**

| Parameter            | Type           | Required | Default     | Description                                                                                                                                     |
| -------------------- | -------------- | -------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Condition            | OTTL Condition | No       | *(empty)*   | An OTTL condition that must evaluate to true for a log to be eligible for deduplication. When empty, every log is eligible.                     |
| Interval             | Integer        | Yes      | `10`        | The time window, in seconds, over which duplicate logs are collapsed. One merged record is emitted per distinct log after the interval elapses. |
| Count Attribute Name | String         | Yes      | `log_count` | The attribute added to the emitted record holding the number of logs that were merged.                                                          |

By default, two logs are considered duplicates when their severity, body, resource attributes, and log attributes all match. Use Included Fields or Excluded Fields to narrow what counts toward that match.

**Field Selection**

| Parameter       | Type            | Required | Default   | Description                                                                                                                                                                                                                                     |
| --------------- | --------------- | -------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Included Fields | OTTL Field Rows | No       | *(empty)* | Restrict matching to only these fields. Each row picks a context (`body` or `attributes`) and a key. Mutually exclusive with Excluded Fields. The whole body cannot be included as a single field.                                              |
| Excluded Fields | OTTL Field Rows | No       | *(empty)* | Ignore these fields when matching, so logs that differ only in these fields still count as duplicates. Each row picks a context (`body` or `attributes`) and a key. Mutually exclusive with Included Fields. The whole body cannot be excluded. |

Only one of Included Fields and Excluded Fields can be set at a time; each hides the other in the UI once it has rows. Nested keys are addressed with dot notation (`src.ip`); a literal dot inside a key name is escaped as `\.` (`log\.file\.name`).

**Advanced**

| Parameter | Type   | Required | Default | Description                                                                                                                                                                      |
| --------- | ------ | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Timezone  | String | Yes      | `UTC`   | The timezone used to localize the `first_observed_timestamp` and `last_observed_timestamp` attributes on the emitted record. Accepts an IANA name such as `America/Los_Angeles`. |

### Examples

#### Deduplicate on a longer interval, ignoring noisy fields

A chatty service emits the same log many times per minute, differing only in a per-event timestamp and the source filename. Collapse them on a 60-second window, exclude the fields that vary, and write the merged count to a custom attribute.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: deduplicate-logs
spec:
  type: log_dedup_v2
  parameters:
    - name: interval
      value: 60
    - name: log_count_attribute
      value: dedup_count
    - name: timezone
      value: America/Los_Angeles
    - name: exclude_fields
      value:
        - context: body
          key: timestamp
        - context: attributes
          key: log.file.name
        - context: attributes
          key: src.ip
```

#### Deduplicate only matching logs on a specific field

Apply the processor only to logs from a single service, and treat two logs as duplicates when their `event_id` attribute matches, ignoring everything else.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: deduplicate-logs
spec:
  type: log_dedup_v2
  parameters:
    - name: condition
      value: attributes["service.name"] == "checkout"
    - name: interval
      value: 30
    - name: log_count_attribute
      value: log_count
    - name: include_fields
      value:
        - context: attributes
          key: event_id
```

### Configuration Tips

* Included Fields and Excluded Fields are mutually exclusive. Excluded Fields are dropped from the emitted record, so do not exclude a field you still need downstream.
* The processor buffers one in-memory entry per distinct log seen during the interval. A long interval combined with high log cardinality (many fields contributing to the match key) grows memory use, since every unique combination is tracked until the window closes. Narrow the match set with Included Fields, or shorten the interval, when memory is a concern.
* A larger interval increases deduplication efficiency but delays emission, since a merged record is held until its window elapses. Pick the interval against how much output latency the pipeline can tolerate.

### Troubleshooting

#### Logs are not being deduplicated

Symptoms: the count attribute stays at 1 and duplicate-looking logs pass through unchanged.

Solutions:

1. Confirm the logs are genuinely identical across the full default match set (severity, body, resource attributes, log attributes). A single differing attribute or resource attribute makes them distinct.
2. If only some fields should matter, set Included Fields to those fields, or use Excluded Fields to ignore the fields that vary between otherwise-identical logs.
3. Check the Condition. If set, only logs where it evaluates true are eligible for deduplication.

#### Output is delayed or arrives in bursts

Symptoms: logs appear downstream in clumps rather than steadily.

Solutions:

1. This is expected. Merged records are emitted only when the interval closes, so a 60-second interval can delay a record by up to 60 seconds.
2. Lower the Interval to reduce latency, accepting fewer merges per window.

#### Memory usage grows under high log volume

Symptoms: collector memory climbs while the processor is active on a high-throughput pipeline.

Solutions:

1. Reduce the Interval so buffered entries are flushed sooner.
2. Use Included Fields to shrink the match key to the fields that actually distinguish duplicates, lowering the number of distinct entries held per window.

### Standalone Processor

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: deduplicate-logs
spec:
  type: log_dedup_v2
  parameters:
    - name: condition
      value: attributes["service.name"] == "checkout"
    - name: interval
      value: 60
    - name: log_count_attribute
      value: dedup_count
    - name: timezone
      value: America/Los_Angeles
    - name: exclude_fields
      value:
        - context: body
          key: timestamp
        - context: attributes
          key: log.file.name
        - context: attributes
          key: src.ip
```

### Related Resources

* [Log Deduplication Processor — OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/logdedupprocessor)
* [OTTL Log Context](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottllog/README.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.bindplane.com/integrations/processors/deduplicate-logs.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.