> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/processors/detect-log-type.md).

# Detect Log Type

Identify the type of every log, and record it to an attribute.

The Detect Log Type processor identifies what produced each log and writes the result to the `log_type` attribute, for example `cisco_asa_firewall`, `aws_cloudtrail` or `nginx`. It ships with built-in matchers for **900+** log types, which Bindplane keeps up to date. You can add your own matchers for in-house applications. Logs that match nothing are labeled `unknown`. The processor only adds attributes. It never drops, parses, or rewrites records.

Use it to route, filter, or standardize logs by type when one source carries many kinds of logs, such as a syslog or HTTP endpoint that receives traffic from many devices.

{% hint style="info" %}
This processor is in **Beta** stability.
{% endhint %}

### Supported Telemetry Types

| Metrics | Logs | Traces |
| ------- | ---- | ------ |
|         | ✓    |        |

### How It Works

1. **Fingerprint.** The processor computes a fingerprint of each log body that captures its structure and ignores its values. Two JSON logs with the same keys share a fingerprint whatever their values are. So do two syslog lines from the same program with the same message shape, or two access log lines for the same request path.
2. **Check the cache.** If the fingerprint has been seen before, the cached log type is applied. Most logs take this path. See [Performance](#performance) for timings.
3. **Run the matchers.** For a new fingerprint, the matchers are tested against the log body in priority order, and the first match wins. Your custom matchers are always tested before the built-in ones. The result is cached under the fingerprint, so the matchers run once per structure, not once per log.
4. **Write the attribute.** The detected type is written to the `log_type` attribute. Logs that match no matcher, and logs shorter than 10 characters, get `log_type: unknown`.

Matchers test the raw log body as a string. A structured (map) body is serialized before it is tested.

#### Example

An Okta System Log event arrives:

```json
{"eventType": "user.session.start", "displayMessage": "User login to Okta", "actor": {"alternateId": "alice@example.com"}}
```

1. The processor fingerprints it. This is the first log with this set of keys, so the fingerprint isn't in the cache.
2. The processor tests the matchers in priority order. The okta matcher is the first to match.
3. The processor caches okta under the fingerprint and writes `log_type: okta`.

A second Okta event arrives for a different user and action:

```json
{"eventType": "user.session.end", "displayMessage": "User logout from Okta", "actor": {"alternateId": "bob@example.com"}}
```

1. The processor fingerprints it. The keys match the first event, so the fingerprint is the same.
2. The fingerprint is in the cache, so the processor writes `log_type: okta` without running any matchers.

### Configuration

<figure><img src="https://1405008107-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgmiOMzBfoNFwmKJFHMcJ%2Fuploads%2FBosleqdRejeZeIOBW5fA%2Fimage.png?alt=media&amp;token=b4e501da-dc9c-45f4-a3b0-4dbf9fcde95c" alt=""><figcaption></figcaption></figure>

#### Templating

| Parameter | Type | Required | Default   | Description                                                                                                                                                                                                                         |
| --------- | ---- | -------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Matchers  | Map  | No       | *(empty)* | Custom matchers, as pairs of a log type name and an [RE2 regular expression](https://github.com/google/re2/wiki/Syntax). They are tested in the order listed, before any built-in matcher. See [Custom Matchers](#custom-matchers). |

**Advanced**

| Parameter                     | Type      | Required | Default        | Description                                                                                                                                                                               |
| ----------------------------- | --------- | -------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Fingerprint Attribute         | String    | No       | *(empty)*      | Attribute to write each log's structure fingerprint to, as a hex string. Leave empty to omit it. Useful for finding the distinct structures in an `unknown` log stream.                   |
| Fetch Matchers From Bindplane | Boolean   | No       | `true`         | Download the built-in matchers from Bindplane when the collector starts. When disabled, only your custom matchers are used.                                                               |
| Matcher Request Timeout       | Duration  | No       | `30s`          | How long the collector keeps asking Bindplane for the built-in matchers after it starts. Set to `0s` to keep asking indefinitely. Only used when Fetch Matchers From Bindplane is `true`. |
| Enable Persistent Storage     | Boolean   | No       | `true`         | Save the downloaded matchers and the fingerprint cache so they survive collector restarts.                                                                                                |
| Persist Interval              | Duration  | No       | `5m`           | How often the fingerprint cache is saved. It is also saved on shutdown. Only used when Enable Persistent Storage is `true`.                                                               |
| Persistent Storage Type       | Extension | Yes      | `file_storage` | The storage extension to save to. Must be a Storage Extension. Only used when Enable Persistent Storage is `true`.                                                                        |

### Built-in Matchers

Built-in matchers cover **900+** log types across security and network devices, cloud services, operating systems, databases, web servers, SaaS applications, and common logging formats. See [Supported Log Types](https://docs.bindplane.com/integrations/processors/detect-log-type/supported-log-types.md) for the full list.

#### How collectors get them

The built-in matchers are not part of the configuration Bindplane sends to the collector. Instead, the processor requests them from Bindplane over the collector's OpAMP connection when the collector starts:

* The request runs in the background and never delays startup. The collector retries every 5 seconds until Bindplane answers or the Matcher Request Timeout passes.
* With persistent storage enabled, the matchers are saved on the collector. On later restarts they are loaded from disk right away, and Bindplane only resends them when a newer set is available.
* Logs processed before the matchers arrive are labeled using whatever matchers the collector already has. On a brand new collector without stored matchers, these first logs are labeled `unknown` unless a custom matcher matches them. When the built-in matchers arrive, the cache is cleared and new logs are detected with the full set. Logs that were already sent keep their original label.

#### Matcher versions

Built-in matchers are versioned as a set. Each release of the processor in Bindplane is pinned to a matcher set version, so the matchers your collectors run only change when you choose to roll out. When a Bindplane upgrade includes new or improved matchers, the configurations that use this processor need a new rollout before their collectors pick up the new set.

### Custom Matchers

Add a custom matcher for any log type the built-in matchers don't cover, or to override one. Each matcher is a **Log Type**, the value written to `log_type`, and a **Regex** that the raw log body must match. Custom matchers are tested in the order you list them, and all of them are tested before the built-in matchers. A custom matcher always takes precedence over a built-in matcher for the same log.

Patterns use Go's [RE2 syntax](https://github.com/google/re2/wiki/Syntax). Lookarounds and backreferences are not supported.

#### Writing good matchers

Match the structure of the log and a signature unique to its source, never a value that changes between logs or environments:

* **Good signatures:** the syslog program name, the CEF or LEEF vendor and product fields, a distinctive combination of JSON keys, a fixed CSV column order, or a message ID such as `%ASA-6-302013`.
* **Bad signatures:** hostnames, usernames, IP addresses and other values. They change between environments and break the matcher.

Write patterns the regex engine can reject quickly. A matcher runs against every new log structure, including logs from other sources:

* **Anchor the pattern with a single leading `^`**, for example `^(?:A|B)`. A pattern written as `^A|^B` is not anchored as a whole and is much slower.
* **Or start the pattern with a distinctive literal**, such as a JSON key: `"eventSource"`. The engine finds a leading literal with a fast substring scan.
* **Avoid a leading `.*`, `\S+`, `\b` or `(?i)`.** They force the engine to try a match at every character of every log.

```
# Anchored syslog header and program name
^<\d{1,3}>\w{3} [ \d]\d \d{2}:\d{2}:\d{2} \S+ myapp\[

# JSON log that starts with a distinctive key
"service":"checkout"

# CSV with a fixed first column
^ORDER_EVENT,
```

### Performance

On a known log structure, the processor costs under a microsecond per log. The real cost comes from new structures, because each one runs the matchers. How fast the processor runs depends mostly on how many distinct structures your logs have.

#### Benchmarks

The processor was measured with 956 matchers against 9,688 sample logs covering every supported log type, with an average length of 360 bytes. The results cover the processor alone, not the rest of the pipeline. Measured on an Apple M4 Pro, with one CPU core unless noted.

| Scenario                                         | Time                         | Throughput per core |
| ------------------------------------------------ | ---------------------------- | ------------------- |
| Known structure, built-in matchers               | 0.7 µs per log               | \~1.4M logs/s       |
| New structure that a built-in matcher identifies | 0.42 ms median, 2.9 ms p99   |                     |
| Every log is a new structure (worst case)        | 0.51 ms per log              | \~1,950 logs/s      |
| Loading the built-in matchers at startup         | 15 ms and \~12 MiB of memory |                     |

When every log is a new structure, each one runs the matchers. The 9,688 sample logs contain 7,977 distinct structures, so the worst case row is close to every log missing the cache. Real log streams usually repeat a small number of structures.

A log that matches nothing is the most expensive case, because every matcher is tested before it falls back to `unknown`. Matcher cost grows with log length, so very long logs cost more on a new structure.

Throughput scales with cores until about four cores share one processor, then levels off. On the test machine it reached 2.3M logs/s on two cores, 3.7M on four and about 2.1M on eight or more. Lookups share one fingerprint cache.

#### Reduce the cost

* **Keep persistent storage enabled.** It is on by default. Without it, every restart and every configuration rollout starts with an empty cache, and every structure is detected again.
* **Add custom matchers for your biggest sources.** This helps most with plain text and access logs, as shown above.
* **Place the processor where it sees only the logs you need to classify.** If a source has a known type, set the attribute with Add Fields instead.
* **Watch for `unknown` logs.** Logs that match nothing cost the most on every new structure. See [Monitoring](#monitoring).

### Persistent Storage

With persistent storage enabled, the processor saves two things to the storage extension:

* **The built-in matchers**, so they are available immediately on restart without waiting for Bindplane.
* **The fingerprint cache**, so known structures don't need to be detected again after a restart. It is saved every Persist Interval and on shutdown.

The saved cache is tied to the exact matchers it was built with. Adding, changing, reordering, or removing a custom matcher, or receiving a new built-in matcher set, discards the saved cache and log types are detected again from scratch.

If the collector crashes, structures detected since the last save are detected again on restart. Lower the Persist Interval to save more often.

### Monitoring

The processor emits these metrics in the collector's internal telemetry:

| Metric                                                   | Description                                                             |
| -------------------------------------------------------- | ----------------------------------------------------------------------- |
| `otelcol_processor_log_type_detection_logs_classified`   | Log records assigned a log type, with a `log_type` attribute.           |
| `otelcol_processor_log_type_detection_logs_unclassified` | Log records labeled `unknown`, including logs too short to fingerprint. |
| `otelcol_processor_log_type_detection_attempts`          | Matcher runs. One per new log structure.                                |
| `otelcol_processor_log_type_detection_attempts_matched`  | Matcher runs that identified a log type.                                |

Use them to answer two questions:

* **How much of my log volume is identified?** Compare `logs_classified` to `logs_unclassified`.
* **How often do new structures show up?** Watch the rate of `attempts`. A rate that stays high after startup means the stream has more structures than the cache holds. See Log structure cardinality.

### Examples

#### Detect log types with the built-in matchers

Label every log with its type using the defaults: built-in matchers fetched from Bindplane, and matchers and cache saved to a file storage extension.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: detect-log-type
spec:
  type: log_type_detection
  parameters: []
```

#### Add matchers for in-house applications

Identify two internal applications. Both are tested before the built-in matchers.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: detect-log-type
spec:
  type: log_type_detection
  parameters:
    - name: matchers
      value:
        - key: checkout_service
          value: '"service":"checkout"'
        - key: billing_batch
          value: '^\d{4}-\d{2}-\d{2}T\S+ billing-batch\['
```

#### Custom matchers only

Skip the built-in matchers and identify only your own log types. Everything else is labeled `unknown`.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: detect-log-type
spec:
  type: log_type_detection
  parameters:
    - name: matchers
      value:
        - key: checkout_service
          value: '"service":"checkout"'
    - name: fetch_matchers_from_bindplane
      value: false
```

#### Route by log type

The `log_type` attribute can drive later processors and routing with an OTTL condition. For example, `attributes["log_type"] == "cisco_asa_firewall"` can be used in a Filter by Condition processor to keep only firewall logs.

### Configuration Tips

* **Place the processor before anything that parses or rewrites the body.** Matchers are written against the raw log as the source sends it. Parsed or reshaped bodies may no longer match.
* **One processor for many sources.** A single instance can classify a mixed stream. Each instance loads its own copy of the built-in matchers, about 12 MiB of memory, so avoid adding one per source when one shared instance will do.
* **Fix wrong labels with a custom matcher.** If a built-in matcher labels a log wrong, add a custom matcher for the correct type. Custom matchers always run first.
* **Track down unknown logs with fingerprints.** Set the Fingerprint Attribute temporarily, then group `unknown` logs by it to find the distinct structures that need a matcher.

### Troubleshooting

#### Every log is labeled unknown

Symptoms: `log_type` is `unknown` on all logs, including logs from supported sources.

Solutions:

1. Check that Fetch Matchers From Bindplane is enabled.
2. Check that the collector is connected to Bindplane. The matchers are fetched over the collector's OpAMP connection. If Bindplane did not answer within the Matcher Request Timeout, the collector stops asking until its next restart. Raise the timeout, or set it to `0s`, for collectors on slow or intermittent connections.
3. On a brand new collector, the first logs are processed before the matchers arrive. Wait a few seconds and check new logs.
4. Check that the processor sees the raw log body. If an earlier processor parsed or rewrote the body, move this processor before it.

#### A log is labeled with the wrong type

Symptoms: logs from one source are labeled as another, similar product.

Solutions:

1. Add a custom matcher for the correct type. It takes precedence over every built-in matcher.
2. Check whether the source wraps its logs, for example a forwarder that adds its own header or JSON envelope. Matchers recognize the formats listed in [Supported Log Types](https://docs.bindplane.com/integrations/processors/detect-log-type/supported-log-types.md).

#### High CPU usage or low throughput

Symptoms: the collector uses noticeably more CPU after adding the processor, or `attempts` keeps rising long after startup.

Solutions:

1. Check the `attempts` rate. A steady rate means new structures keep arriving, usually from plain text or access logs with many distinct words or paths.
2. Add a custom matcher for that source so new structures are cheap to identify. See [writing good matchers](#writing-good-matchers).
3. Check that persistent storage is enabled, so restarts and rollouts don't empty the cache.

#### New built-in matchers aren't used

Symptoms: a log type listed in Supported Log Types is not detected after a Bindplane upgrade.

Solutions:

1. Roll out the configuration. Collectors only pick up a new matcher set through a configuration rollout.
2. Check that the collector was able to reach Bindplane after it restarted. See [Every log is labeled unknown](#every-log-is-labeled-unknown).

### Standalone Processor

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: detect-log-type
spec:
  type: log_type_detection
  parameters:
    - name: matchers
      value:
        - key: checkout_service
          value: '"service":"checkout"'
    - name: fingerprint_field
      value: fingerprint
    - name: fetch_matchers_from_bindplane
      value: true
    - name: opamp_request_timeout
      value: 30s
    - name: enable_persistent_storage
      value: true
    - name: fingerprint_persist_interval
      value: 5m
    - name: persistent_storage_extension
      value:
        type: file_storage
        parameters:
          - name: directory
            value: /var/lib/otel/storage
```

### Related Resources

* [Supported Log Types](https://docs.bindplane.com/integrations/processors/detect-log-type/supported-log-types.md)
* [RE2 regular expression syntax](https://github.com/google/re2/wiki/Syntax)
* [File Storage extension](https://docs.bindplane.com/integrations/extensions/file-storage.md)
* [Redis Storage extension](https://docs.bindplane.com/integrations/extensions/redis-storage.md)
* [Filter by Condition](https://docs.bindplane.com/integrations/processors/filter-by-condition.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.bindplane.com/integrations/processors/detect-log-type.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
