Kafka Cluster

The Kafka Cluster source collects cluster-level metrics from an Apache Kafka deployment. It connects to one or more brokers over the native Kafka protocol and scrapes broker, topic, and consumer-group metrics. It does not read from a JMX endpoint, so no JMX port or metrics JAR is required. Optional SASL, Kerberos, basic, and TLS authentication are supported.

Supported Telemetry Types

Platform	Metrics	Logs	Traces
Linux	✓
Windows	✓
macOS	✓

Prerequisites

• One or more reachable Kafka brokers, addressed as host:port. The collector scrapes the cluster through this broker list.
• Network reachability from the collector to each broker's listener port (default 9092).
• A Kafka client identity the broker accepts. If your cluster requires authentication, you need credentials for one of the supported mechanisms (basic, SASL, or Kerberos) and any TLS material the listener requires.
• If the broker listener enforces TLS, the CA certificate that signed the broker certificates, plus a client certificate and private key when the listener requires mutual TLS.

The source uses Kafka's standard client metadata and metrics APIs, so no broker-side JMX configuration is needed. For background on what Kafka exposes and how, see the Apache Kafka monitoring documentation.

Configuration

Metrics

Parameter	Type	Required	Default	Description
Cluster Name	String	Yes	(empty)	Friendly name used for the resource kafka.cluster.name.
Protocol Version	Enum: 2.2.1, 2.2.0, 2.0.0, 1.0.0 (custom values allowed)	No	2.0.0	The Kafka protocol version to use when communicating with brokers.
Brokers	Strings	Yes	localhost:9092	List of brokers to scrape for metrics, formatted as host:port.
Client ID	String	Yes	otel-metrics-receiver	The consumer client ID that the receiver will use.
Collection Interval	Integer	No	60	How often (seconds) to scrape for metrics. Advanced option.

Authentication

Parameter	Type	Required	Default	Description
Enable Authentication	Boolean	No	false	Enable to authenticate to the brokers. Advanced option. Reveals the authentication parameters below.
Auth Type	Enum: basic, sasl, kerberos, tls	No	basic	Authentication mechanism to use. Advanced option. Shown when Enable Authentication is true. Each value reveals its own credential fields.
Username	String	Yes	(empty)	Basic-auth username. Shown when Enable Authentication is true and Auth Type is basic. Advanced option.
Password	String	Yes	(empty)	Basic-auth password. Shown when Enable Authentication is true and Auth Type is basic. Advanced option.
Username	String	Yes	(empty)	SASL username. Shown when Enable Authentication is true and Auth Type is sasl. Advanced option.
Password	String	Yes	(empty)	SASL password. Shown when Enable Authentication is true and Auth Type is sasl. Advanced option.
Mechanism	Enum: SCRAM-SHA-256, SCRAM-SHA-512, PLAIN	No	SCRAM-SHA-256	SASL mechanism. Shown when Enable Authentication is true and Auth Type is sasl. Advanced option.
Service Name	String	Yes	(empty)	Kerberos service name. Shown when Enable Authentication is true and Auth Type is kerberos. Advanced option.
Realm	String	Yes	(empty)	Kerberos realm. Shown when Enable Authentication is true and Auth Type is kerberos. Advanced option.
Config File	String	Yes	/etc/krb5.conf	Path to the Kerberos configuration file. Shown when Enable Authentication is true and Auth Type is kerberos. Advanced option.
Kerberos Auth Type	Enum: keytab, basic	No	keytab	How to authenticate to Kerberos. Shown when Enable Authentication is true and Auth Type is kerberos. Advanced option.
Keytab File	String	Yes	/etc/security/kafka.keytab	Path to the Kerberos keytab file. Shown when Auth Type is kerberos and Kerberos Auth Type is keytab. Advanced option.
Username	String	Yes	(empty)	Kerberos username. Shown when Auth Type is kerberos and Kerberos Auth Type is basic. Advanced option.
Password	String	Yes	(empty)	Kerberos password. Shown when Auth Type is kerberos and Kerberos Auth Type is basic. Advanced option.

TLS

Parameter	Type	Required	Default	Description
Enable TLS	Boolean	No	false	Whether or not to use TLS. Advanced option. Reveals the TLS parameters below.
Skip TLS Certificate Verification	Boolean	No	false	Enable to skip TLS certificate verification. Shown when Enable TLS is true. Advanced option.
TLS Certificate Authority File	String	No	(empty)	Certificate authority used to validate TLS certificates. Shown when Enable TLS is true. Advanced option.
Mutual TLS Client Certificate File	String	No	(empty)	A TLS certificate used for client authentication. Shown when Enable TLS is true. Advanced option.
TLS Client Private Key File	String	No	(empty)	A TLS private key used for client authentication. Shown when Enable TLS is true. Advanced option.

Examples

Scraping a multi-broker cluster with SASL/SCRAM

This source scrapes a three-broker cluster, authenticates with SASL using SCRAM-SHA-512, and tags every metric with the cluster name prod-east.

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: kafka-cluster-prod-east
spec:
  type: kafka_cluster
  parameters:
    - name: cluster_name
      value: prod-east
    - name: brokers
      value:
        - broker-1.example.com:9092
        - broker-2.example.com:9092
        - broker-3.example.com:9092
    - name: protocol_version
      value: "2.2.1"
    - name: enable_auth
      value: true
    - name: auth_type
      value: sasl
    - name: sasl_username
      value: monitoring
    - name: sasl_password
      value: REDACTED
    - name: sasl_mechanism
      value: SCRAM-SHA-512

Configuration Tips

• Set a meaningful Cluster Name per deployment. Its value is written to the kafka.cluster.name resource attribute, which is how you distinguish metrics from multiple Kafka clusters in the same pipeline.
• List multiple Brokers for resiliency. The receiver uses the list for cluster metadata discovery, so additional entries keep collection working if one broker is unreachable.
• Pin the Protocol Version to a value your brokers support. The field is creatable, so you can enter a version that is not in the dropdown if your cluster runs a newer release.

Troubleshooting

No metrics appear and the collector logs a connection error

Symptoms: The collector reports connection refused or dial timeouts to the broker address, and no kafka.* metrics are produced.

Solution: Confirm the Brokers list uses reachable host:port values and that the collector host can reach each broker's listener port (default 9092). Check firewall and security-group rules between the collector and the brokers.

Authentication or authorization failures

Symptoms: The collector logs SASL, Kerberos, or authorization errors and fails to fetch metadata.

Solution: Verify the credentials for the selected Auth Type and that the account is permitted to read cluster, topic, and consumer-group metadata. For SASL, confirm the Mechanism matches the broker's configured mechanism. For Kerberos, confirm the Config File, Realm, and Service Name are correct and that the keytab or username/password is valid.

TLS handshake failures

Symptoms: The collector logs a TLS handshake or certificate verification error when connecting to the brokers.

Solution: Ensure Enable TLS matches the broker listener. Provide the TLS Certificate Authority File that signed the broker certificates, and a client certificate and key if the listener requires mutual TLS. As a temporary diagnostic only, Skip TLS Certificate Verification bypasses verification; do not leave it enabled in production.

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: kafka-cluster
spec:
  type: kafka_cluster
  parameters:
    - name: cluster_name
      value: my-kafka-cluster
    - name: brokers
      value:
        - localhost:9092
    - name: protocol_version
      value: "2.0.0"
    - name: client_id
      value: otel-metrics-receiver
    - name: collection_interval
      value: 60

Related Resources

• Apache Kafka monitoring documentation
• OpenTelemetry Kafka Metrics Receiver (kafkametricsreceiver)