> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/processors/lookup-fields.md).

# Lookup Fields

Enriches telemetry by matching the value of a field against an external lookup source. For each record, the processor reads the chosen field, looks it up in a CSV or JSON file, a Redis server, an HTTP API, or a PuppetDB or OpenVoxDB server, and on a match adds the other fields from the matching record to the same context (Attributes, Body, or Resource).

{% hint style="info" %}
Some sources and settings need a newer collector. See [Collector Version Requirements](#collector-version-requirements).
{% endhint %}

### Supported Telemetry Types

| Platform           | Metrics | Logs | Traces |
| ------------------ | ------- | ---- | ------ |
| Linux              | ✓       | ✓    | ✓      |
| Windows            | ✓       | ✓    | ✓      |
| macOS              | ✓       | ✓    | ✓      |
| Kubernetes Node    | ✓       | ✓    | ✓      |
| Kubernetes Cluster | ✓       | ✓    | ✓      |
| Kubernetes Gateway | ✓       | ✓    | ✓      |
| OpenShift Node     | ✓       | ✓    | ✓      |
| OpenShift Cluster  | ✓       | ✓    | ✓      |
| OpenShift Gateway  | ✓       | ✓    | ✓      |

Select one or more signals; the same lookup configuration enriches every selected signal.

### Collector Version Requirements

| Feature                                                       | Minimum Bindplane OTel Collector |
| ------------------------------------------------------------- | -------------------------------- |
| CSV source                                                    | Any                              |
| Redis and API sources, Enable Cache, Cache TTL, Persist Cache | v1.100.0                         |
| Reload Interval, Cache Max Entries                            | v1.107.0                         |
| JSON and PuppetDB sources                                     | v1.112.0                         |

An older collector ignores settings it doesn't support, and rejects the configuration when the selected source isn't supported.

### Configuration

#### Basic Configuration

<figure><img src="https://1405008107-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgmiOMzBfoNFwmKJFHMcJ%2Fuploads%2Fgit-blob-b2c7975f8580abcc2cad4eb0b7c9d0229a0b39e5%2Fintegrations-processors-lookup-fields-image-1.png?alt=media" alt="Bindplane docs - Lookup Fields - image 1"><figcaption></figcaption></figure>

**Selection**

| Parameter             | Type               | Required | Default    | Description                                                                                                                                                                                       |
| --------------------- | ------------------ | -------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Choose Telemetry Type | Telemetry Selector | Yes      | Logs       | The signals this instance runs on. Select Logs, Metrics, and/or Traces.                                                                                                                           |
| Context               | Enum               | No       | Attributes | Where the source field is read from, and where matching fields are added. One of Attributes, Body, or Resource.                                                                                   |
| Field                 | OTTL Field         | Yes      | —          | The field to look up in the selected context. A lookup runs when this field's name and value match a record in the source. Enter the field name (e.g. `host.name` or `ip`), not a full OTTL path. |

**Lookup**

| Parameter       | Type   | Required | Default   | Description                                                                                                                                                                                                                                                                                                      |
| --------------- | ------ | -------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Source Type     | Enum   | Yes      | CSV       | The data source used for lookups. One of CSV (a file), JSON (a file), Redis (a Redis server), API (an HTTP endpoint), or PuppetDB (a PuppetDB or OpenVoxDB server).                                                                                                                                              |
| CSV File Path   | String | Yes \*   | —         | Path to the CSV file containing lookup values. The file must be accessible on the collector host and include the `.csv` extension. Applies when Source Type is CSV.                                                                                                                                              |
| JSON File Path  | String | Yes \*   | —         | Path to the JSON file containing lookup values. The file must be accessible on the collector host. Applies when Source Type is JSON.                                                                                                                                                                             |
| Reload Interval | String | No       | *(empty)* | How often the source is re-checked for changes (e.g. `60s`, `5m`). An unchanged CSV or JSON file is skipped without re-reading it, and a PuppetDB inventory is re-fetched each time. Empty uses the processor default of 60s, or 30m for PuppetDB. Advanced; applies when Source Type is CSV, JSON, or PuppetDB. |

\* CSV File Path is required when Source Type is CSV, and JSON File Path is required when Source Type is JSON.

The JSON file is one object whose keys are lookup values and whose values are objects of fields to add. Each key can carry its own set of fields, and only the fields present for the matched key are added. Values are added as strings: a number keeps its source text, a boolean becomes `true` or `false`, a nested object or array is added as compact JSON, and a `null` field is skipped.

#### Redis

These parameters apply when Source Type is Redis.

| Parameter      | Type    | Required | Default   | Description                                                     |
| -------------- | ------- | -------- | --------- | --------------------------------------------------------------- |
| Redis Host     | String  | Yes      | localhost | Hostname or IP address of the Redis server.                     |
| Redis Port     | Integer | Yes      | 6379      | TCP port of the Redis server (1–65535).                         |
| Redis Username | String  | No       | *(empty)* | Username for Redis authentication.                              |
| Redis Password | String  | No       | *(empty)* | Password for Redis authentication. Stored as a sensitive value. |
| Redis Database | Integer | No       | 0         | The Redis database number.                                      |
| Key Prefix     | String  | No       | *(empty)* | Prefix prepended to lookup keys, joined with a colon (`:`).     |

**Redis Advanced**

These parameters apply when Source Type is Redis.

| Parameter            | Type    | Required | Default   | Description                                                                                              |
| -------------------- | ------- | -------- | --------- | -------------------------------------------------------------------------------------------------------- |
| Enable TLS           | Boolean | No       | false     | Enable TLS for Redis connections.                                                                        |
| Redis Dial Timeout   | String  | No       | *(empty)* | Bounds the initial TCP/TLS dial to Redis (e.g. `2s`, `500ms`). Empty uses the processor default of `2s`. |
| Redis Lookup Timeout | String  | No       | *(empty)* | Bounds each Redis lookup call (e.g. `5s`, `1s`). Empty uses the processor default of `5s`.               |

#### API

These parameters apply when Source Type is API. On each lookup the field value is substituted into the URL, the endpoint is queried, and fields from the JSON response are added to the context.

A `5xx`, `408`, or `429` response is retried with exponential backoff; other `4xx` responses are treated as deterministic and are not retried. The processor reads at most 1 MiB of each response body (a fixed, non-configurable cap).

| Parameter        | Type   | Required | Default   | Description                                                                                                                                                            |
| ---------------- | ------ | -------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| API URL          | String | Yes      | —         | URL to query for lookups. The tokens `$fieldValue`, `${fieldValue}`, `$key`, or `${key}` are replaced with the URL-encoded lookup key.                                 |
| HTTP Method      | Enum   | No       | GET       | The HTTP method used for lookup requests. One of GET or POST.                                                                                                          |
| Request Headers  | Map    | No       | *(empty)* | Headers to send with each lookup request, such as an authorization token.                                                                                              |
| Response Mapping | Map    | No       | *(empty)* | Maps output field names to dotted JSON paths in the response (e.g. `host` → `data.hostname`). When empty, the top-level response object is flattened into the context. |

**API Advanced**

These parameters apply when Source Type is API.

| Parameter           | Type    | Required | Default   | Description                                                                                                                          |
| ------------------- | ------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Request Timeout     | String  | No       | *(empty)* | Per-request HTTP timeout for a single attempt (e.g. `10s`). Empty uses the processor default of `10s`.                               |
| Lookup Timeout      | String  | No       | *(empty)* | Overall bound for one lookup including all retries and backoff (e.g. `5s`). Empty uses the processor default of `5s`.                |
| Max Retries         | Integer | No       | 3         | Total attempts (initial request plus retries) for transient failures. Non-retryable statuses (400, 401, 403, 404) abort immediately. |
| Initial Retry Delay | String  | No       | *(empty)* | Backoff before the first retry (e.g. `100ms`). Empty uses the processor default of `100ms`.                                          |
| Retry Multiplier    | Integer | No       | 2         | Multiplier applied to the backoff between retries.                                                                                   |

#### PuppetDB

These parameters apply when Source Type is PuppetDB. The lookup value is matched against each node's certname. The same settings work with PuppetDB and OpenVoxDB, which share the same query API.

On each reload the processor fetches the selected facts for every node and holds them in memory, so lookups never call the server. A failed reload is logged and the previous inventory stays in use.

| Parameter         | Type   | Required | Default                        | Description                                                                                                                                                              |
| ----------------- | ------ | -------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| PuppetDB Endpoint | String | Yes      | —                              | URL of the PuppetDB or OpenVoxDB server (e.g. `https://puppetdb.example.com:8081`).                                                                                      |
| Facts             | Map    | Yes      | Seven common facts<sup>1</sup> | Sets each field from a fact path (e.g. `os.family` from `facts.os.family`). A path is `environment`, or starts with `facts.` or `trusted.`. Only listed facts are added. |

1. *`puppet.environment` from `environment`, `puppet.kernel` from `facts.kernel`, `puppet.os.name` from `facts.os.name`, `puppet.os.family` from `facts.os.family`, `puppet.os.release` from `facts.os.release.full`, `puppet.os.architecture` from `facts.os.architecture`, and `puppet.virtual` from `facts.virtual`. Puppet agents report these facts on Linux, Windows, and macOS.*

**PuppetDB Advanced**

These parameters apply when Source Type is PuppetDB.

| Parameter                         | Type    | Required | Default   | Description                                                                                                                                                                 |
| --------------------------------- | ------- | -------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Match Short Hostname              | Boolean | No       | false     | Also match a lookup value against the first label of each certname, so `web01` matches `web01.example.com`. A short name that more than one certname shares is not matched. |
| Node Query                        | String  | No       | *(empty)* | A PuppetDB query in AST form that limits which nodes are loaded (e.g. `["=", "facts.os.family", "RedHat"]`).                                                                |
| Request Timeout                   | String  | No       | 30s       | Timeout for each request to the server.                                                                                                                                     |
| Page Size                         | Integer | No       | 1000      | Number of nodes fetched per request.                                                                                                                                        |
| Max Nodes                         | Integer | No       | 100000    | Maximum number of nodes loaded. A reload that would exceed it fails, and the previous inventory stays in use.                                                               |
| Minimum TLS Version               | Enum    | No       | 1.2       | The minimum TLS version to support. One of 1.3, 1.2, 1.1, or 1.0.                                                                                                           |
| Skip TLS Certificate Verification | Boolean | No       | false     | Skip verification of the server's certificate.                                                                                                                              |
| Server Name Override              | String  | No       | *(empty)* | The server name to verify the certificate against. Set it to the server's certname when the endpoint uses an address or alias the certificate doesn't list.                 |
| TLS Certificate Authority File    | String  | No       | *(empty)* | Certificate authority file used to validate the server's certificate.                                                                                                       |
| Mutual TLS                        | Boolean | No       | false     | Present a client certificate to the server. PuppetDB 9 and OpenVoxDB require one by default.                                                                                |
| TLS Client Certificate File       | String  | No       | *(empty)* | Path to the client certificate. Applies when Mutual TLS is on.                                                                                                              |
| TLS Client Private Key            | String  | No       | *(empty)* | Path to the client private key. Applies when Mutual TLS is on.                                                                                                              |

#### Caching

These parameters apply when Source Type is Redis or API. A CSV or JSON file and a PuppetDB inventory are already held in memory, so they are looked up directly and the cache does not apply.

| Parameter         | Type    | Required | Default      | Description                                                                                                                                                   |
| ----------------- | ------- | -------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Enable Cache      | Boolean | No       | true         | Cache lookup results locally to reduce external calls.                                                                                                        |
| Cache TTL         | String  | No       | 5m           | How long cached entries remain valid (e.g. `5m`, `1h`).                                                                                                       |
| Cache Max Entries | Integer | No       | 100000       | Maximum entries held by the in-memory cache. On overflow, expired entries are evicted first, then arbitrary entries. Ignored when a storage extension is set. |
| Persist Cache     | Boolean | No       | false        | Back the cache with a storage extension so cached results survive collector restarts. When off, the cache is in-memory and discarded on restart.              |
| Storage Extension | Enum    | Yes †    | File Storage | The storage extension that backs the cache. Applies when Persist Cache is on.                                                                                 |

† Storage Extension is required when Persist Cache is on.

### Examples

#### Enrich metrics from a CSV by host name

This instance looks up the `host.name` resource value of incoming metrics against a CSV file. When a row's `host.name` matches, the other columns from that row (here `region` and `env`) are added to the same context. For example, a metric with `host.name` of `MacBook-Pro-4.local` gains `region=us-east` and `env=prod`.

Example CSV:

```csv
host.name,region,env
MacBook-Pro-4.local,us-east,prod
MacBook-Pro-3.local,us-west,dev
MacBook-Pro-2.local,us-central,dev
MacBook-Pro-1.local,us-central,prod
```

#### Enrich from a JSON file by host name

<figure><img src="https://1405008107-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgmiOMzBfoNFwmKJFHMcJ%2Fuploads%2Fgit-blob-29e884e363b1c71532146e436a63cdd68fe25432%2Fintegrations-processors-lookup-fields-image-4.png?alt=media" alt="Bindplane docs - Lookup Fields - image 4"><figcaption></figcaption></figure>

This instance sets Source Type to JSON and looks up the `host.name` resource value of logs, metrics, and traces in `/etc/otel/hosts.json`. Unlike a CSV, each host can carry a different set of fields. Here `web-01` gains `env`, `region`, and `owner`, while `db-01` gains only `env`.

```json
{
  "web-01": {"env": "prod", "region": "us-east", "owner": "web-team"},
  "db-01": {"env": "prod"}
}
```

#### Enrich from PuppetDB over mutual TLS

<figure><img src="https://1405008107-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgmiOMzBfoNFwmKJFHMcJ%2Fuploads%2Fgit-blob-b62144bd5a461b584fa9393cd42d188eef0ec886%2Fintegrations-processors-lookup-fields-image-5.png?alt=media" alt="Bindplane docs - Lookup Fields - image 5"><figcaption></figcaption></figure>

This instance sets Source Type to PuppetDB and adds Puppet facts to logs and metrics by host name. Facts keeps the seven default facts and adds `puppet.role` from `trusted.extensions.pp_role`. Match Short Hostname is on, so telemetry that reports `web01` matches the certname `web01.example.com`. Node Query loads only Linux nodes.

The server is PuppetDB 9, which only accepts clients that present a certificate signed by the Puppet CA. Mutual TLS is on, and the collector presents its own Puppet agent certificate and key from `/etc/puppetlabs/puppet/ssl/`. Reload Interval is `30m`, matching the Puppet agent's default run interval.

#### Look up from Redis with a persistent cache

<figure><img src="https://1405008107-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgmiOMzBfoNFwmKJFHMcJ%2Fuploads%2Fgit-blob-f05f0c81f153c843b18f7e84fabd8f24bb93409a%2Fintegrations-processors-lookup-fields-image-2.png?alt=media" alt="Bindplane docs - Lookup Fields - image 2"><figcaption></figcaption></figure>

This instance sets Source Type to Redis and looks up the `user_id` attribute for incoming logs. With Key Prefix `user`, a log whose `user_id` is `web-01` reads the Redis key `user:web-01`, and the returned fields (such as `region` and `env`) are added to the log attributes. The advanced parameters are exercised here: TLS is on with a Redis Dial Timeout of `3s` and a Redis Lookup Timeout of `2s`; the cache holds up to 50,000 entries for `10m`; and Persist Cache is on with a File Storage extension, so cached lookups survive a collector restart.

The processor first tries `HGETALL` against the key. If that returns nothing, it falls back to `GET` and parses the value as a JSON object of strings. Store lookup data as either a Redis hash or a JSON string:

```
# Redis hash
HSET user:web-01 region us-east env prod

# JSON string
SET user:web-01 '{"region":"us-east","env":"prod"}'
```

#### Look up from an HTTP API

<figure><img src="https://1405008107-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgmiOMzBfoNFwmKJFHMcJ%2Fuploads%2Fgit-blob-1ca9a4a1c4a6f3a77dc4af60138b9d369ce2b600%2Fintegrations-processors-lookup-fields-image-3.png?alt=media" alt="Bindplane docs - Lookup Fields - image 3"><figcaption></figcaption></figure>

This instance sets Source Type to API and enriches incoming logs from a CMDB endpoint. The `host.name` resource value is substituted into the URL `https://cmdb.internal/hosts/${fieldValue}`, an `Authorization` header carries a bearer token, and Response Mapping pulls `team` from `data.owner.team` and `env` from `data.environment` in the JSON response. Max Retries is `4` with an Initial Retry Delay of `200ms`, and lookup results are cached for `15m`.

### Configuration Tips

* Enter Field as a plain field name (e.g. `host.name`), not an OTTL path expression. The processor reads and writes it within the selected Context.
* For a CSV or JSON source, the processor re-checks the file on the Reload Interval and re-reads it only when it has changed, so you can update lookup data without restarting the collector.
* For PuppetDB, every reload fetches the whole selected inventory, and every collector running the processor sends its own requests. Keep the Reload Interval near the Puppet agent's run interval rather than lowering it to seconds. List only the facts you need, since facts can hold sensitive data such as cloud metadata or SSH host keys.
* For Redis, set Key Prefix to namespace your lookup keys; it is joined to the field value with a colon.
* For an API source, keep secrets out of the config by referencing an environment variable in a header (e.g. `Bearer ${env:CMDB_TOKEN}`).
* The cache applies to Redis and API only. To keep cached lookups across restarts, turn on Persist Cache and choose a [File Storage](https://docs.bindplane.com/integrations/extensions/file-storage.md) or [Redis Storage](https://docs.bindplane.com/integrations/extensions/redis-storage.md) extension.

### Troubleshooting

#### Fields are not added after a lookup

Symptoms: matching records exist in the source, but no new fields appear on the telemetry.

Solutions:

1. Confirm the Context is the one that actually holds the source field, and that Field is a plain name (not an OTTL path).
2. For CSV, verify the file path is correct and readable on the collector host and that a header matches the Field name. For Redis, confirm the key (Key Prefix plus the field value) exists. For API, confirm the URL resolves and the Response Mapping paths match the JSON the endpoint returns.

#### PuppetDB rejects the collector

Symptoms: the collector logs a failed PuppetDB load that mentions the certificate allowlist, or a TLS handshake error.

Solutions:

1. Turn on Mutual TLS and point TLS Client Certificate File and TLS Client Private Key at a certificate signed by the Puppet CA. On a Puppet-managed host, the agent's own certificate under `/etc/puppetlabs/puppet/ssl/` works.
2. Ask the Puppet administrator to add that certificate's certname to the server's `certificate-allowlist` file.
3. If the endpoint uses an IP address or an alias, set Server Name Override to the server's certname.

#### Lookup results don't show in Live Preview

Symptoms: enrichment isn't visible when previewing the configuration.

Solutions: Live Preview doesn't run lookups for any source type. Roll out the configuration, then use "View Recent Telemetry" on the collector page or inspect a downstream processor node to verify.

#### The collector fails to start with a Redis source

Symptoms: the collector exits at startup instead of failing on the first lookup.

Solutions: On startup the processor sends a `PING` to Redis within the dial timeout. Confirm Redis Host, Redis Port, and credentials are correct and the server is reachable. Adjust Redis Dial Timeout if the network is slow.

#### API lookups are slow or time out

Symptoms: the pipeline stalls, or lookups fail under load.

Solutions: Bound each call with Request Timeout and cap the whole lookup (including retries) with Lookup Timeout, so a slow endpoint can't exceed Request Timeout × Max Retries. Keep the cache on to avoid repeated calls for the same key.

### Standalone Processor

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  name: lookup-fields
spec:
  type: lookup_fields
  parameters:
    - name: telemetry
      value:
        - Logs
    - name: source_type
      value: Redis
    - name: context
      value: Attributes
    - name: field
      value: user_id
    - name: redis_host
      value: localhost
    - name: redis_port
      value: 6379
    - name: redis_key_prefix
      value: user
    - name: cache_enabled
      value: true
    - name: cache_ttl
      value: 10m
    - name: cache_max_entries
      value: 50000
    - name: redis_dial_timeout
      value: 2s
    - name: redis_lookup_timeout
      value: 5s
```

### Related Resources

* [PuppetDB query API](https://www.puppet.com/docs/puppetdb/8/api/query/v4/overview)
* [PuppetDB certificate allowlist](https://www.puppet.com/docs/puppetdb/8/configure#certificate-allowlist)
* [OpenVoxDB](https://voxpupuli.org/openvox/)
* [Redis HGETALL command](https://redis.io/docs/latest/commands/hgetall/)
* [Redis GET command](https://redis.io/docs/latest/commands/get/)

### Bindplane Resources

These processors enrich and reshape telemetry alongside Lookup Fields. Chain them to parse, normalize, and add context to your data:

* [Add Fields](https://docs.bindplane.com/integrations/processors/add-fields.md)
* [Parse JSON](https://docs.bindplane.com/integrations/processors/parse-json.md)
* [Parse CSV](https://docs.bindplane.com/integrations/processors/parse-csv.md)
* [Geo IP](https://docs.bindplane.com/integrations/processors/geoip.md)
* [File Storage](https://docs.bindplane.com/integrations/extensions/file-storage.md)
* [Redis Storage](https://docs.bindplane.com/integrations/extensions/redis-storage.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.bindplane.com/integrations/processors/lookup-fields.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
