> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/sources/kafka-node.md).

# Kafka Node

The Kafka Node source collects telemetry from an individual Apache Kafka node. It scrapes per-node JMX metrics for the broker, consumer, and producer subsystems using the OpenTelemetry JMX metrics gathering JAR, and it tails the node's local log files (server, controller, state-change, and log-cleaner logs). Choose Logs, Metrics, or both.

### Supported Telemetry

| Platform | Metrics | Logs | Traces |
| -------- | ------- | ---- | ------ |
| Linux    | ✓       | ✓    |        |
| Windows  | ✓       | ✓    |        |
| macOS    | ✓       | ✓    |        |

### Prerequisites

**For metrics (JMX):**

* A reachable JMX/RMI endpoint on the Kafka node. Kafka disables remote JMX by default; enable it by setting the `JMX_PORT` environment variable before starting Kafka, and secure it for production with `KAFKA_JMX_OPTS`. See the [Apache Kafka monitoring documentation](https://kafka.apache.org/documentation/#monitoring).
* The collector must be able to reach the configured JMX address and port (default `localhost:9999`).
* The OpenTelemetry JMX metrics gathering JAR on the collector host, at the path given by the JMX Metrics Collection Jar Path parameter (default `/opt/opentelemetry-java-contrib-jmx-metrics.jar`).
* A Java runtime on the collector host, which the JMX metrics JAR requires.

**For logs:**

* The Kafka node writing the log files you want to collect (server, controller, state-change, log-cleaner).
* The configured log file path(s) must exist and the collector must have read access to them. Defaults assume a `/home/kafka/kafka/logs/` layout; adjust to match your installation.

### Configuration

<figure><img src="/files/i4P5CTG7zEWPyG60IebR" alt="Bindplane docs - Kafka Node - image 1"><figcaption></figcaption></figure>

**Telemetry Type**

| Parameter             | Type               | Required | Default               | Description                                                      |
| --------------------- | ------------------ | -------- | --------------------- | ---------------------------------------------------------------- |
| Choose Telemetry Type | Telemetry Selector | No       | `["Logs", "Metrics"]` | Which signals this source collects. Valid values: Logs, Metrics. |

**Logs**

These parameters apply when Logs is selected.

| Parameter                 | Type    | Required | Default                                    | Description                                                                          |
| ------------------------- | ------- | -------- | ------------------------------------------ | ------------------------------------------------------------------------------------ |
| Server Logs               | Boolean | No       | `true`                                     | Collect Kafka server logs.                                                           |
| Server File Path(s)       | Strings | No       | `/home/kafka/kafka/logs/server.log`        | File paths to tail for server logs. Applies when Server Logs is enabled.             |
| Controller Logs           | Boolean | No       | `true`                                     | Collect Kafka controller logs.                                                       |
| Controller File Path(s)   | Strings | No       | `/home/kafka/kafka/logs/controller.log`    | File paths to tail for controller logs. Applies when Controller Logs is enabled.     |
| State Change Logs         | Boolean | No       | `true`                                     | Collect Kafka state-change logs.                                                     |
| Stage Change File Path(s) | Strings | No       | `/home/kafka/kafka/logs/state-change.log`  | File paths to tail for state-change logs. Applies when State Change Logs is enabled. |
| Log Cleaner Logs          | Boolean | No       | `true`                                     | Collect Kafka log-cleaner logs.                                                      |
| Log Cleaner File Path(s)  | Strings | No       | `/home/kafka/kafka/logs/state-cleaner.log` | File paths to tail for log-cleaner logs. Applies when Log Cleaner Logs is enabled.   |

**Metrics**

These parameters apply when Metrics is selected.

| Parameter | Type    | Required | Default     | Description                                       |
| --------- | ------- | -------- | ----------- | ------------------------------------------------- |
| Address   | String  | No       | `localhost` | IP address or hostname to scrape for JMX metrics. |
| Port      | Integer | No       | `9999`      | Port to scrape for JMX metrics.                   |

**Advanced**

| Parameter                       | Type                 | Required | Default                                           | Description                                                                  |
| ------------------------------- | -------------------- | -------- | ------------------------------------------------- | ---------------------------------------------------------------------------- |
| Start At                        | Enum: beginning, end | No       | `end`                                             | Start reading logs from 'beginning' or 'end'. Applies when Logs is selected. |
| Parse                           | Boolean              | No       | `true`                                            | Parse the log fields into structured data. Applies when Logs is selected.    |
| JMX Metrics Collection Jar Path | String               | No       | `/opt/opentelemetry-java-contrib-jmx-metrics.jar` | Full path to the JMX metrics JAR. Applies when Metrics is selected.          |
| Collection Interval             | Integer              | No       | `60`                                              | How often (seconds) to scrape for metrics. Applies when Metrics is selected. |

### Examples

#### Metrics only from a remote broker

Scrape JMX metrics from a Kafka node listening on a non-default JMX port, every 30 seconds, and skip log collection.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: kafka-node
spec:
  type: kafka_node
  parameters:
    - name: telemetry_types
      value:
        - Metrics
    - name: address
      value: 10.0.0.21
    - name: port
      value: 8999
    - name: collection_interval
      value: 30
```

### Configuration Tips

* The source emits three JMX receivers per node (broker, consumer, and producer target systems), all scraping the same JMX endpoint. They share the Address, Port, JAR path, and Collection Interval settings.
* Kafka does not expose remote JMX by default. If metrics never arrive, confirm `JMX_PORT` is set on the Kafka process and the port matches the Port parameter.
* Log file paths default to a `/home/kafka/kafka/logs/` layout. Set the path parameters to match your actual Kafka installation directory.

### Troubleshooting

#### Connection refused on the JMX port

Symptoms: metrics never appear and the collector logs a connection error against the JMX endpoint.

Solutions:

1. Confirm remote JMX is enabled on the Kafka node (`JMX_PORT` set before startup) and that the Address and Port parameters match.
2. Verify the collector host can reach the JMX/RMI port through any firewalls or security groups.

#### No metrics, JAR error

Symptoms: the collector reports it cannot launch or find the JMX metrics JAR.

Solutions:

1. Confirm the OpenTelemetry JMX metrics JAR exists at the JMX Metrics Collection Jar Path on the collector host.
2. Verify a Java runtime is installed on the collector host.

#### No logs collected

Symptoms: log collection is enabled but no log records arrive.

Solutions:

1. Confirm the configured file path(s) exist and the collector has read access to them.
2. If the source was just added and you expect existing log content, set Start At to `beginning`.

### Standalone Source

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: kafka-node
spec:
  type: kafka_node
  parameters:
    - name: telemetry_types
      value:
        - Logs
        - Metrics
    - name: enable_server_log
      value: true
    - name: server_log_path
      value:
        - /home/kafka/kafka/logs/server.log
    - name: enable_controller_log
      value: true
    - name: controller_log_path
      value:
        - /home/kafka/kafka/logs/controller.log
    - name: enable_state_change_log
      value: true
    - name: state_change_log_path
      value:
        - /home/kafka/kafka/logs/state-change.log
    - name: enable_log_cleaner_log
      value: true
    - name: log_cleaner_log_path
      value:
        - /home/kafka/kafka/logs/state-cleaner.log
    - name: start_at
      value: end
    - name: parse
      value: true
    - name: address
      value: localhost
    - name: port
      value: 9999
    - name: jar_path
      value: /opt/opentelemetry-java-contrib-jmx-metrics.jar
    - name: collection_interval
      value: 60
```

### Related Resources

* [Apache Kafka monitoring documentation](https://kafka.apache.org/documentation/#monitoring)
* [OpenTelemetry JMX Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jmxreceiver)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.bindplane.com/integrations/sources/kafka-node.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.