Deduplicate Logs

The Deduplicate Logs processor watches the log stream over a fixed interval and merges identical records into one. The emitted record carries a count attribute with the number of logs that were collapsed, plus first- and last-observed timestamps. It operates on logs only.

Supported Telemetry Types

Metrics	Logs	Traces
	✓

Configuration

Matching

Parameter	Type	Required	Default	Description
Condition	OTTL Condition	No	(empty)	An OTTL condition that must evaluate to true for a log to be eligible for deduplication. When empty, every log is eligible.
Interval	Integer	Yes	10	The time window, in seconds, over which duplicate logs are collapsed. One merged record is emitted per distinct log after the interval elapses.
Count Attribute Name	String	Yes	log_count	The attribute added to the emitted record holding the number of logs that were merged.

By default, two logs are considered duplicates when their severity, body, resource attributes, and log attributes all match. Use Included Fields or Excluded Fields to narrow what counts toward that match.

Field Selection

Parameter	Type	Required	Default	Description
Included Fields	OTTL Field Rows	No	(empty)	Restrict matching to only these fields. Each row picks a context (body or attributes) and a key. Mutually exclusive with Excluded Fields. The whole body cannot be included as a single field.
Excluded Fields	OTTL Field Rows	No	(empty)	Ignore these fields when matching, so logs that differ only in these fields still count as duplicates. Each row picks a context (body or attributes) and a key. Mutually exclusive with Included Fields. The whole body cannot be excluded.

Only one of Included Fields and Excluded Fields can be set at a time; each hides the other in the UI once it has rows. Nested keys are addressed with dot notation (src.ip); a literal dot inside a key name is escaped as \. (log\.file\.name).

Advanced

Parameter	Type	Required	Default	Description
Timezone	String	Yes	UTC	The timezone used to localize the first_observed_timestamp and last_observed_timestamp attributes on the emitted record. Accepts an IANA name such as America/Los_Angeles.

Examples

Deduplicate on a longer interval, ignoring noisy fields

A chatty service emits the same log many times per minute, differing only in a per-event timestamp and the source filename. Collapse them on a 60-second window, exclude the fields that vary, and write the merged count to a custom attribute.

apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: deduplicate-logs
spec:
  type: log_dedup_v2
  parameters:
    - name: interval
      value: 60
    - name: log_count_attribute
      value: dedup_count
    - name: timezone
      value: America/Los_Angeles
    - name: exclude_fields
      value:
        - context: body
          key: timestamp
        - context: attributes
          key: log.file.name
        - context: attributes
          key: src.ip

Deduplicate only matching logs on a specific field

Apply the processor only to logs from a single service, and treat two logs as duplicates when their event_id attribute matches, ignoring everything else.

apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: deduplicate-logs
spec:
  type: log_dedup_v2
  parameters:
    - name: condition
      value: attributes["service.name"] == "checkout"
    - name: interval
      value: 30
    - name: log_count_attribute
      value: log_count
    - name: include_fields
      value:
        - context: attributes
          key: event_id

Configuration Tips

• Included Fields and Excluded Fields are mutually exclusive. Excluded Fields are dropped from the emitted record, so do not exclude a field you still need downstream.
• The processor buffers one in-memory entry per distinct log seen during the interval. A long interval combined with high log cardinality (many fields contributing to the match key) grows memory use, since every unique combination is tracked until the window closes. Narrow the match set with Included Fields, or shorten the interval, when memory is a concern.
• A larger interval increases deduplication efficiency but delays emission, since a merged record is held until its window elapses. Pick the interval against how much output latency the pipeline can tolerate.

Troubleshooting

Logs are not being deduplicated

Symptoms: the count attribute stays at 1 and duplicate-looking logs pass through unchanged.

Solutions:

1. Confirm the logs are genuinely identical across the full default match set (severity, body, resource attributes, log attributes). A single differing attribute or resource attribute makes them distinct.
2. If only some fields should matter, set Included Fields to those fields, or use Excluded Fields to ignore the fields that vary between otherwise-identical logs.
3. Check the Condition. If set, only logs where it evaluates true are eligible for deduplication.

Output is delayed or arrives in bursts

Symptoms: logs appear downstream in clumps rather than steadily.

Solutions:

1. This is expected. Merged records are emitted only when the interval closes, so a 60-second interval can delay a record by up to 60 seconds.
2. Lower the Interval to reduce latency, accepting fewer merges per window.

Memory usage grows under high log volume

Symptoms: collector memory climbs while the processor is active on a high-throughput pipeline.

Solutions:

1. Reduce the Interval so buffered entries are flushed sooner.
2. Use Included Fields to shrink the match key to the fields that actually distinguish duplicates, lowering the number of distinct entries held per window.

Standalone Processor

apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: deduplicate-logs
spec:
  type: log_dedup_v2
  parameters:
    - name: condition
      value: attributes["service.name"] == "checkout"
    - name: interval
      value: 60
    - name: log_count_attribute
      value: dedup_count
    - name: timezone
      value: America/Los_Angeles
    - name: exclude_fields
      value:
        - context: body
          key: timestamp
        - context: attributes
          key: log.file.name
        - context: attributes
          key: src.ip

Related Resources

• Log Deduplication Processor — OpenTelemetry Collector Contrib
• OTTL Log Context