Kafka

The Kafka destination publishes telemetry from a Bindplane pipeline to one or more Apache Kafka topics. It exports logs, metrics, and traces, with a separate topic and encoding configurable per signal. Authentication (basic, SASL, Kerberos, or TLS) and TLS transport security are supported for secured clusters.

Supported Telemetry Types

Metrics	Logs	Traces
✓	✓	✓

Prerequisites

You need a reachable Kafka cluster and at least one topic for each signal you intend to export.

• Reachable brokers. One or more broker addresses in host:port form that the collector can connect to. The default is localhost:9092.
• Topics. A topic for each enabled signal. Kafka can auto-create topics when the broker is configured with auto.create.topics.enable=true; otherwise create the topics ahead of time. Defaults are otlp_logs (logs), otlp_metrics (metrics), and otlp_spans (traces).
• Authentication credentials (if the cluster requires them). Depending on the cluster, this is a basic username/password, a SASL username/password and mechanism (SCRAM-SHA-256, SCRAM-SHA-512, or PLAIN), a Kerberos principal with a keytab or password plus a krb5.conf, or a TLS client certificate.
• TLS material (if the cluster requires TLS). A CA certificate to validate the broker, and optionally a client certificate and key for mutual TLS.

For broker security configuration, see the Apache Kafka security documentation.

Configuration

Connection

Parameter	Type	Required	Default	Description
Choose Telemetry Type	Telemetry Selector	No	Logs, Metrics, Traces	The signals this destination exports.
Protocol Version	Enum	No	2.0.0	The Kafka protocol version to use when communicating with brokers. Selectable values are 3.6.0, 3.2.0, 2.2.1, 2.2.0, 2.0.0, and 1.0.0; a custom version can be entered if it is not listed.
Brokers	Strings	Yes	localhost:9092	List of brokers (host:port) to connect to when publishing logs, metrics, and traces.
Timeout	Integer	No	5	Timeout in seconds for every attempt to publish data to the backend.

Logs

These parameters apply when Logs is selected in Choose Telemetry Type.

Parameter	Type	Required	Default	Description
Log Topic	String	No	otlp_logs	The name of the topic to publish logs to.
Log Encoding	Enum	No	otlp_proto	The encoding used when publishing logs. One of otlp_proto, otlp_json, or raw.

Metrics

These parameters apply when Metrics is selected in Choose Telemetry Type.

Parameter	Type	Required	Default	Description
Metric Topic	String	No	otlp_metrics	The name of the topic to publish metrics to.
Metric Encoding	Enum	No	otlp_proto	The encoding used when publishing metrics. One of otlp_proto or otlp_json.

Traces

These parameters apply when Traces is selected in Choose Telemetry Type.

Parameter	Type	Required	Default	Description
Trace Topic	String	No	otlp_spans	The name of the topic to publish traces to.
Trace Encoding	Enum	No	otlp_proto	The encoding used when publishing traces. One of otlp_proto, otlp_json, jaeger_proto, jaeger_json, zipkin_proto, or zipkin_json.

Advanced

Parameter	Type	Required	Default	Description
Compression	Enum	No	gzip	The compression algorithm to use when publishing data. One of none, gzip, snappy, or lz4.
Max Message Bytes	Integer	No	1000000	The maximum size of a message in bytes.
Required Acknowledgements	Integer	No	1	The number of acknowledgements the producer requires the leader to have received before considering a request complete.
Flush Max Messages	Integer	No	0	The maximum number of messages the producer will send in a single batch. 0 lets the producer decide.
Drop Raw Copy	Boolean	No	true	When enabled, the raw copy of the log stored in log.record.original is dropped before publishing. Logs only.

Authentication (Advanced)

Parameter	Type	Required	Default	Description
Enable Authentication	Boolean	No	false	Enable authentication to the Kafka cluster.
Auth Type	Enum	No	basic	The authentication method. One of basic, sasl, kerberos, or tls. Shown when Enable Authentication is true.
Username (basic)	String	Yes	(empty)	Basic auth username. Shown when Auth Type is basic.
Password (basic)	String	Yes	(empty)	Basic auth password. Sensitive. Shown when Auth Type is basic.
Username (SASL)	String	Yes	(empty)	SASL username. Shown when Auth Type is sasl.
Password (SASL)	String	Yes	(empty)	SASL password. Sensitive. Shown when Auth Type is sasl.
Mechanism (SASL)	Enum	No	SCRAM-SHA-256	SASL mechanism. One of SCRAM-SHA-256, SCRAM-SHA-512, or PLAIN. Shown when Auth Type is sasl.
Service Name (Kerberos)	String	Yes	(empty)	Kerberos service name. Shown when Auth Type is kerberos.
Realm (Kerberos)	String	Yes	(empty)	Kerberos realm. Shown when Auth Type is kerberos.
Config File (Kerberos)	String	Yes	/etc/krb5.conf	Path to the Kerberos configuration file. Shown when Auth Type is kerberos.
Kerberos Auth Type	Enum	No	keytab	How the collector authenticates to Kerberos. One of keytab or basic. Shown when Auth Type is kerberos.
Keytab File (Kerberos)	String	Yes	/etc/security/kafka.keytab	Path to the keytab file. Shown when Kerberos Auth Type is keytab.
Username (Kerberos)	String	Yes	(empty)	Kerberos username. Shown when Kerberos Auth Type is basic.
Password (Kerberos)	String	Yes	(empty)	Kerberos password. Sensitive. Shown when Kerberos Auth Type is basic.

TLS (Advanced)

Parameter	Type	Required	Default	Description
Enable TLS	Boolean	No	false	Whether or not to use TLS.
Skip TLS Certificate Verification	Boolean	No	false	Enable to skip TLS certificate verification. Shown when Enable TLS is true.
TLS Certificate Authority File	String	No	(empty)	Certificate authority used to validate the broker's TLS certificate. Shown when Enable TLS is true and certificate verification is not skipped.
TLS Client Certificate File	String	No	(empty)	A TLS certificate used for client authentication (mutual TLS). Shown when Enable TLS is true.
TLS Client Private Key File	String	No	(empty)	A TLS private key used for client authentication (mutual TLS). Shown when Enable TLS is true.
Server Name Override	String	No	(empty)	Indicates the name of the server requested by the client in order to support virtual hosting. Shown when Enable TLS is true.

Retry on Failure (Advanced)

Parameter	Type	Required	Default	Description
Enable Retry on Failure	Boolean	No	true	Attempt to resend telemetry that failed to be transmitted to the destination.
Initial interval	Integer	No	5	Time in seconds to wait after the first failure before retrying.
Max interval	Integer	No	30	The upper bound in seconds on backoff.
Max elapsed time	Integer	No	300	The maximum time in seconds spent trying to send a batch before giving up, to avoid a never-ending retry loop.

Sending Queue (Advanced)

Parameter	Type	Required	Default	Description
Enable Sending Queue	Boolean	No	true	Buffer telemetry in memory before sending to help avoid data loss during a temporary network outage.
Number of Consumers	Integer	No	10	Number of consumers that dequeue batches. Shown when Enable Sending Queue is true.
Queue Size	Integer	No	5000	Maximum number of batches kept in memory before dropping. Shown when Enable Sending Queue is true.
Enable Persistent Queuing	Boolean	No	true	Buffer telemetry to disk before sending to help avoid data loss during network outages or collector restarts. Shown when Enable Sending Queue is true.
Persistent Queue Storage	Extension	Yes	File storage	The storage extension used for the persistent queue. Defaults to file storage at ${OIQ_OTEL_COLLECTOR_HOME}/storage. Shown when Enable Sending Queue and Enable Persistent Queuing are true.

Examples

Publish all three signals to a secured cluster over SASL and TLS

This destination exports logs, metrics, and traces to three brokers, authenticates with SASL (SCRAM-SHA-512), and validates the broker with a CA certificate over TLS. Snappy compression is used to reduce network usage.

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  name: kafka
spec:
  type: kafka_otlp_destination_v2
  parameters:
    - name: telemetry_types
      value:
        - Logs
        - Metrics
        - Traces
    - name: protocol_version
      value: "3.6.0"
    - name: brokers
      value:
        - broker-1.kafka.internal:9093
        - broker-2.kafka.internal:9093
        - broker-3.kafka.internal:9093
    - name: log_topic
      value: otlp_logs
    - name: metric_topic
      value: otlp_metrics
    - name: trace_topic
      value: otlp_spans
    - name: compression
      value: snappy
    - name: enable_auth
      value: true
    - name: auth_type
      value: sasl
    - name: sasl_username
      value: my-sasl-user
    - name: sasl_password
      value: "${SASL_PASSWORD}"
    - name: sasl_mechanism
      value: SCRAM-SHA-512
    - name: enable_tls
      value: true
    - name: tls_ca_file
      value: /etc/otel/certs/ca.crt

Configuration Tips

• Each signal publishes to its own topic. Set Log Topic, Metric Topic, and Trace Topic independently so consumers can subscribe per signal.
• Use otlp_proto encoding (the default) when the downstream consumer reads OTLP. Choose jaeger_* or zipkin_* trace encodings, or raw log encoding, only when the consumer expects that format.
• Compression (gzip, snappy, or lz4) reduces broker network usage at some CPU cost. snappy and lz4 trade a lower compression ratio for lower CPU than gzip.
• Leave the sending queue and persistent queue enabled to buffer telemetry to disk during broker or network outages, which avoids dropping data on collector restarts.

Troubleshooting

Cannot connect to brokers

Symptoms: the collector logs connection refused, dial timeout, or broker resolution errors, and no data reaches Kafka.

Solutions:

1. Confirm each broker address is reachable from the collector host in host:port form.
2. Verify the Protocol Version is compatible with the cluster. Set it explicitly if the cluster runs an older or newer broker than the default.

Authentication or TLS handshake fails

Symptoms: the collector logs SASL authentication failures, a TLS handshake error, or a certificate verification error.

Solutions:

1. Confirm Auth Type matches the cluster (basic, SASL mechanism, Kerberos, or TLS) and that credentials are correct.
2. For TLS, provide the correct CA file. Use Skip TLS Certificate Verification only for testing, and set Server Name Override when the certificate's name differs from the broker address.

Telemetry is dropped under load

Symptoms: gaps in data at the consumer, or the collector logs that the sending queue is full.

Solutions:

1. Increase Queue Size and Number of Consumers so the producer can keep up with throughput.
2. Keep the persistent queue enabled so buffered telemetry survives collector restarts, and confirm the storage path is writable.

Standalone Destination

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  name: kafka
spec:
  type: kafka_otlp_destination_v2
  parameters:
    - name: telemetry_types
      value:
        - Logs
        - Metrics
        - Traces
    - name: brokers
      value:
        - localhost:9092
    - name: log_topic
      value: otlp_logs
    - name: metric_topic
      value: otlp_metrics
    - name: trace_topic
      value: otlp_spans

Related Resources

• OpenTelemetry Kafka Exporter
• Apache Kafka security documentation