Kafka Node

The Kafka Node source collects telemetry from an individual Apache Kafka node. It scrapes per-node JMX metrics for the broker, consumer, and producer subsystems using the OpenTelemetry JMX metrics gathering JAR, and it tails the node's local log files (server, controller, state-change, and log-cleaner logs). Choose Logs, Metrics, or both.

Supported Telemetry Types

Platform	Metrics	Logs	Traces
Linux	✓	✓
Windows	✓	✓
macOS	✓	✓

Prerequisites

For metrics (JMX):

• A reachable JMX/RMI endpoint on the Kafka node. Kafka disables remote JMX by default; enable it by setting the JMX_PORT environment variable before starting Kafka, and secure it for production with KAFKA_JMX_OPTS. See the Apache Kafka monitoring documentation.
• The collector must be able to reach the configured JMX address and port (default localhost:9999).
• The OpenTelemetry JMX metrics gathering JAR on the collector host, at the path given by the JMX Metrics Collection Jar Path parameter (default /opt/opentelemetry-java-contrib-jmx-metrics.jar).
• A Java runtime on the collector host, which the JMX metrics JAR requires.

For logs:

• The Kafka node writing the log files you want to collect (server, controller, state-change, log-cleaner).
• The configured log file path(s) must exist and the collector must have read access to them. Defaults assume a /home/kafka/kafka/logs/ layout; adjust to match your installation.

Configuration

Telemetry Type

Parameter	Type	Required	Default	Description
Choose Telemetry Type	Telemetry Selector	No	["Logs", "Metrics"]	Which signals this source collects. Valid values: Logs, Metrics.

Logs

These parameters apply when Logs is selected.

Parameter	Type	Required	Default	Description
Server Logs	Boolean	No	true	Collect Kafka server logs.
Server File Path(s)	Strings	No	/home/kafka/kafka/logs/server.log	File paths to tail for server logs. Applies when Server Logs is enabled.
Controller Logs	Boolean	No	true	Collect Kafka controller logs.
Controller File Path(s)	Strings	No	/home/kafka/kafka/logs/controller.log	File paths to tail for controller logs. Applies when Controller Logs is enabled.
State Change Logs	Boolean	No	true	Collect Kafka state-change logs.
Stage Change File Path(s)	Strings	No	/home/kafka/kafka/logs/state-change.log	File paths to tail for state-change logs. Applies when State Change Logs is enabled.
Log Cleaner Logs	Boolean	No	true	Collect Kafka log-cleaner logs.
Log Cleaner File Path(s)	Strings	No	/home/kafka/kafka/logs/state-cleaner.log	File paths to tail for log-cleaner logs. Applies when Log Cleaner Logs is enabled.

Metrics

These parameters apply when Metrics is selected.

Parameter	Type	Required	Default	Description
Address	String	No	localhost	IP address or hostname to scrape for JMX metrics.
Port	Integer	No	9999	Port to scrape for JMX metrics.

Advanced

Parameter	Type	Required	Default	Description
Start At	Enum: beginning, end	No	end	Start reading logs from 'beginning' or 'end'. Applies when Logs is selected.
Parse	Boolean	No	true	Parse the log fields into structured data. Applies when Logs is selected.
JMX Metrics Collection Jar Path	String	No	/opt/opentelemetry-java-contrib-jmx-metrics.jar	Full path to the JMX metrics JAR. Applies when Metrics is selected.
Collection Interval	Integer	No	60	How often (seconds) to scrape for metrics. Applies when Metrics is selected.

Examples

Metrics only from a remote broker

Scrape JMX metrics from a Kafka node listening on a non-default JMX port, every 30 seconds, and skip log collection.

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: kafka-node
spec:
  type: kafka_node
  parameters:
    - name: telemetry_types
      value:
        - Metrics
    - name: address
      value: 10.0.0.21
    - name: port
      value: 8999
    - name: collection_interval
      value: 30

Configuration Tips

• The source emits three JMX receivers per node (broker, consumer, and producer target systems), all scraping the same JMX endpoint. They share the Address, Port, JAR path, and Collection Interval settings.
• Kafka does not expose remote JMX by default. If metrics never arrive, confirm JMX_PORT is set on the Kafka process and the port matches the Port parameter.
• Log file paths default to a /home/kafka/kafka/logs/ layout. Set the path parameters to match your actual Kafka installation directory.

Troubleshooting

Connection refused on the JMX port

Symptoms: metrics never appear and the collector logs a connection error against the JMX endpoint.

Solutions:

1. Confirm remote JMX is enabled on the Kafka node (JMX_PORT set before startup) and that the Address and Port parameters match.
2. Verify the collector host can reach the JMX/RMI port through any firewalls or security groups.

No metrics, JAR error

Symptoms: the collector reports it cannot launch or find the JMX metrics JAR.

Solutions:

1. Confirm the OpenTelemetry JMX metrics JAR exists at the JMX Metrics Collection Jar Path on the collector host.
2. Verify a Java runtime is installed on the collector host.

No logs collected

Symptoms: log collection is enabled but no log records arrive.

Solutions:

1. Confirm the configured file path(s) exist and the collector has read access to them.
2. If the source was just added and you expect existing log content, set Start At to beginning.

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: kafka-node
spec:
  type: kafka_node
  parameters:
    - name: telemetry_types
      value:
        - Logs
        - Metrics
    - name: enable_server_log
      value: true
    - name: server_log_path
      value:
        - /home/kafka/kafka/logs/server.log
    - name: enable_controller_log
      value: true
    - name: controller_log_path
      value:
        - /home/kafka/kafka/logs/controller.log
    - name: enable_state_change_log
      value: true
    - name: state_change_log_path
      value:
        - /home/kafka/kafka/logs/state-change.log
    - name: enable_log_cleaner_log
      value: true
    - name: log_cleaner_log_path
      value:
        - /home/kafka/kafka/logs/state-cleaner.log
    - name: start_at
      value: end
    - name: parse
      value: true
    - name: address
      value: localhost
    - name: port
      value: 9999
    - name: jar_path
      value: /opt/opentelemetry-java-contrib-jmx-metrics.jar
    - name: collection_interval
      value: 60

Related Resources

• Apache Kafka monitoring documentation
• OpenTelemetry JMX Receiver