> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/destinations/kafka.md). # Kafka The Kafka destination publishes telemetry from a Bindplane pipeline to one or more Apache Kafka topics. It exports logs, metrics, and traces, with a separate topic and encoding configurable per signal. Authentication (basic, SASL, Kerberos, or TLS) and TLS transport security are supported for secured clusters. ### Supported Telemetry | Metrics | Logs | Traces | | ------- | ---- | ------ | | ✓ | ✓ | ✓ | ### Prerequisites You need a reachable Kafka cluster and at least one topic for each signal you intend to export. * **Reachable brokers.** One or more broker addresses in `host:port` form that the collector can connect to. The default is `localhost:9092`. * **Topics.** A topic for each enabled signal. Kafka can auto-create topics when the broker is configured with `auto.create.topics.enable=true`; otherwise create the topics ahead of time. Defaults are `otlp_logs` (logs), `otlp_metrics` (metrics), and `otlp_spans` (traces). * **Authentication credentials (if the cluster requires them).** Depending on the cluster, this is a basic username/password, a SASL username/password and mechanism (`SCRAM-SHA-256`, `SCRAM-SHA-512`, or `PLAIN`), a Kerberos principal with a keytab or password plus a `krb5.conf`, or a TLS client certificate. * **TLS material (if the cluster requires TLS).** A CA certificate to validate the broker, and optionally a client certificate and key for mutual TLS. For broker security configuration, see the [Apache Kafka security documentation](https://kafka.apache.org/documentation/#security). ### Configuration
Bindplane docs - Kafka - image 1
**Connection** | Parameter | Type | Required | Default | Description | | --------------------- | ------------------ | -------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Choose Telemetry Type | Telemetry Selector | No | `Logs`, `Metrics`, `Traces` | The signals this destination exports. | | Protocol Version | Enum | No | `2.0.0` | The Kafka protocol version to use when communicating with brokers. Selectable values are `3.6.0`, `3.2.0`, `2.2.1`, `2.2.0`, `2.0.0`, and `1.0.0`; a custom version can be entered if it is not listed. | | Brokers | Strings | Yes | `localhost:9092` | List of brokers (`host:port`) to connect to when publishing logs, metrics, and traces. | | Timeout | Integer | No | `5` | Timeout in seconds for every attempt to publish data to the backend. | **Logs** These parameters apply when `Logs` is selected in Choose Telemetry Type. | Parameter | Type | Required | Default | Description | | ------------ | ------ | -------- | ------------ | ----------------------------------------------------------------------------------- | | Log Topic | String | No | `otlp_logs` | The name of the topic to publish logs to. | | Log Encoding | Enum | No | `otlp_proto` | The encoding used when publishing logs. One of `otlp_proto`, `otlp_json`, or `raw`. | **Metrics** These parameters apply when `Metrics` is selected in Choose Telemetry Type. | Parameter | Type | Required | Default | Description | | --------------- | ------ | -------- | -------------- | ------------------------------------------------------------------------------ | | Metric Topic | String | No | `otlp_metrics` | The name of the topic to publish metrics to. | | Metric Encoding | Enum | No | `otlp_proto` | The encoding used when publishing metrics. One of `otlp_proto` or `otlp_json`. | **Traces** These parameters apply when `Traces` is selected in Choose Telemetry Type. | Parameter | Type | Required | Default | Description | | -------------- | ------ | -------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | Trace Topic | String | No | `otlp_spans` | The name of the topic to publish traces to. | | Trace Encoding | Enum | No | `otlp_proto` | The encoding used when publishing traces. One of `otlp_proto`, `otlp_json`, `jaeger_proto`, `jaeger_json`, `zipkin_proto`, or `zipkin_json`. | **Advanced** | Parameter | Type | Required | Default | Description | | ------------------------- | ------- | -------- | --------- | ----------------------------------------------------------------------------------------------------------------------- | | Compression | Enum | No | `gzip` | The compression algorithm to use when publishing data. One of `none`, `gzip`, `snappy`, or `lz4`. | | Max Message Bytes | Integer | No | `1000000` | The maximum size of a message in bytes. | | Required Acknowledgements | Integer | No | `1` | The number of acknowledgements the producer requires the leader to have received before considering a request complete. | | Flush Max Messages | Integer | No | `0` | The maximum number of messages the producer will send in a single batch. `0` lets the producer decide. | | Drop Raw Copy | Boolean | No | `true` | When enabled, the raw copy of the log stored in `log.record.original` is dropped before publishing. Logs only. | **Authentication (Advanced)** | Parameter | Type | Required | Default | Description | | ----------------------- | ------- | -------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------- | | Enable Authentication | Boolean | No | `false` | Enable authentication to the Kafka cluster. | | Auth Type | Enum | No | `basic` | The authentication method. One of `basic`, `sasl`, `kerberos`, or `tls`. Shown when Enable Authentication is `true`. | | Username (basic) | String | Yes | *(empty)* | Basic auth username. Shown when Auth Type is `basic`. | | Password (basic) | String | Yes | *(empty)* | Basic auth password. Sensitive. Shown when Auth Type is `basic`. | | Username (SASL) | String | Yes | *(empty)* | SASL username. Shown when Auth Type is `sasl`. | | Password (SASL) | String | Yes | *(empty)* | SASL password. Sensitive. Shown when Auth Type is `sasl`. | | Mechanism (SASL) | Enum | No | `SCRAM-SHA-256` | SASL mechanism. One of `SCRAM-SHA-256`, `SCRAM-SHA-512`, or `PLAIN`. Shown when Auth Type is `sasl`. | | Service Name (Kerberos) | String | Yes | *(empty)* | Kerberos service name. Shown when Auth Type is `kerberos`. | | Realm (Kerberos) | String | Yes | *(empty)* | Kerberos realm. Shown when Auth Type is `kerberos`. | | Config File (Kerberos) | String | Yes | `/etc/krb5.conf` | Path to the Kerberos configuration file. Shown when Auth Type is `kerberos`. | | Kerberos Auth Type | Enum | No | `keytab` | How the collector authenticates to Kerberos. One of `keytab` or `basic`. Shown when Auth Type is `kerberos`. | | Keytab File (Kerberos) | String | Yes | `/etc/security/kafka.keytab` | Path to the keytab file. Shown when Kerberos Auth Type is `keytab`. | | Username (Kerberos) | String | Yes | *(empty)* | Kerberos username. Shown when Kerberos Auth Type is `basic`. | | Password (Kerberos) | String | Yes | *(empty)* | Kerberos password. Sensitive. Shown when Kerberos Auth Type is `basic`. | **TLS (Advanced)** | Parameter | Type | Required | Default | Description | | --------------------------------- | ------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | Enable TLS | Boolean | No | `false` | Whether or not to use TLS. | | Skip TLS Certificate Verification | Boolean | No | `false` | Enable to skip TLS certificate verification. Shown when Enable TLS is `true`. | | TLS Certificate Authority File | String | No | *(empty)* | Certificate authority used to validate the broker's TLS certificate. Shown when Enable TLS is `true` and certificate verification is not skipped. | | TLS Client Certificate File | String | No | *(empty)* | A TLS certificate used for client authentication (mutual TLS). Shown when Enable TLS is `true`. | | TLS Client Private Key File | String | No | *(empty)* | A TLS private key used for client authentication (mutual TLS). Shown when Enable TLS is `true`. | | Server Name Override | String | No | *(empty)* | Indicates the name of the server requested by the client in order to support virtual hosting. Shown when Enable TLS is `true`. | **Retry on Failure (Advanced)** | Parameter | Type | Required | Default | Description | | ----------------------- | ------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------- | | Enable Retry on Failure | Boolean | No | `true` | Attempt to resend telemetry that failed to be transmitted to the destination. | | Initial interval | Integer | No | `5` | Time in seconds to wait after the first failure before retrying. | | Max interval | Integer | No | `30` | The upper bound in seconds on backoff. | | Max elapsed time | Integer | No | `300` | The maximum time in seconds spent trying to send a batch before giving up, to avoid a never-ending retry loop. | **Sending Queue (Advanced)** | Parameter | Type | Required | Default | Description | | ------------------------- | --------- | -------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Enable Sending Queue | Boolean | No | `true` | Buffer telemetry in memory before sending to help avoid data loss during a temporary network outage. | | Number of Consumers | Integer | No | `10` | Number of consumers that dequeue batches. Shown when Enable Sending Queue is `true`. | | Queue Size | Integer | No | `5000` | Maximum number of batches kept in memory before dropping. Shown when Enable Sending Queue is `true`. | | Enable Persistent Queuing | Boolean | No | `true` | Buffer telemetry to disk before sending to help avoid data loss during network outages or collector restarts. Shown when Enable Sending Queue is `true`. | | Persistent Queue Storage | Extension | Yes | File storage | The storage extension used for the persistent queue. Defaults to file storage at `${OIQ_OTEL_COLLECTOR_HOME}/storage`. Shown when Enable Sending Queue and Enable Persistent Queuing are `true`. | ### Examples #### Publish all three signals to a secured cluster over SASL and TLS This destination exports logs, metrics, and traces to three brokers, authenticates with SASL (`SCRAM-SHA-512`), and validates the broker with a CA certificate over TLS. Snappy compression is used to reduce network usage. ```yaml apiVersion: bindplane.observiq.com/v1 kind: Destination metadata: name: kafka spec: type: kafka_otlp_destination_v2 parameters: - name: telemetry_types value: - Logs - Metrics - Traces - name: protocol_version value: "3.6.0" - name: brokers value: - broker-1.kafka.internal:9093 - broker-2.kafka.internal:9093 - broker-3.kafka.internal:9093 - name: log_topic value: otlp_logs - name: metric_topic value: otlp_metrics - name: trace_topic value: otlp_spans - name: compression value: snappy - name: enable_auth value: true - name: auth_type value: sasl - name: sasl_username value: my-sasl-user - name: sasl_password value: "${SASL_PASSWORD}" - name: sasl_mechanism value: SCRAM-SHA-512 - name: enable_tls value: true - name: tls_ca_file value: /etc/otel/certs/ca.crt ``` ### Configuration Tips * Each signal publishes to its own topic. Set Log Topic, Metric Topic, and Trace Topic independently so consumers can subscribe per signal. * Use `otlp_proto` encoding (the default) when the downstream consumer reads OTLP. Choose `jaeger_*` or `zipkin_*` trace encodings, or `raw` log encoding, only when the consumer expects that format. * Compression (`gzip`, `snappy`, or `lz4`) reduces broker network usage at some CPU cost. `snappy` and `lz4` trade a lower compression ratio for lower CPU than `gzip`. * Leave the sending queue and persistent queue enabled to buffer telemetry to disk during broker or network outages, which avoids dropping data on collector restarts. ### Troubleshooting #### Cannot connect to brokers Symptoms: the collector logs connection refused, dial timeout, or broker resolution errors, and no data reaches Kafka. Solutions: 1. Confirm each broker address is reachable from the collector host in `host:port` form. 2. Verify the Protocol Version is compatible with the cluster. Set it explicitly if the cluster runs an older or newer broker than the default. #### Authentication or TLS handshake fails Symptoms: the collector logs SASL authentication failures, a TLS handshake error, or a certificate verification error. Solutions: 1. Confirm Auth Type matches the cluster (basic, SASL mechanism, Kerberos, or TLS) and that credentials are correct. 2. For TLS, provide the correct CA file. Use Skip TLS Certificate Verification only for testing, and set Server Name Override when the certificate's name differs from the broker address. #### Telemetry is dropped under load Symptoms: gaps in data at the consumer, or the collector logs that the sending queue is full. Solutions: 1. Increase Queue Size and Number of Consumers so the producer can keep up with throughput. 2. Keep the persistent queue enabled so buffered telemetry survives collector restarts, and confirm the storage path is writable. ### Standalone Destination ```yaml apiVersion: bindplane.observiq.com/v1 kind: Destination metadata: name: kafka spec: type: kafka_otlp_destination_v2 parameters: - name: telemetry_types value: - Logs - Metrics - Traces - name: brokers value: - localhost:9092 - name: log_topic value: otlp_logs - name: metric_topic value: otlp_metrics - name: trace_topic value: otlp_spans ``` ### Related Resources * [OpenTelemetry Kafka Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/kafkaexporter/README.md) * [Apache Kafka security documentation](https://kafka.apache.org/documentation/#security) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.bindplane.com/integrations/destinations/kafka.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.