> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/sources/hadoop.md).

# Hadoop

The Apache Hadoop source collects metrics over JMX from Hadoop daemons (NameNode, DataNode, Resource Manager) and tails their on-disk log files. Metrics are scraped through the OpenTelemetry Java JMX metrics gatherer using the `hadoop` target system; logs are read from the daemon log paths and optionally parsed into structured fields.

### Supported Telemetry

| Platform | Metrics | Logs | Traces |
| -------- | ------- | ---- | ------ |
| macOS    | ✓       | ✓    |        |
| Linux    | ✓       | ✓    |        |
| Windows  | ✓       | ✓    |        |

### Prerequisites

For metrics, the Hadoop daemon you target must have **JMX remote enabled** and a reachable JMX/RMI port (the source defaults to port `8004`). The collector host needs network reachability to that address and port. JVM JMX remote access is typically enabled by setting `com.sun.management.jmxremote*` system properties (and authentication/TLS where required) in the daemon's `HADOOP_*_OPTS` environment variables. Metrics are also published through the Metrics2 framework as standard JMX MBeans. See the official Apache Hadoop metrics documentation for details: [Apache Hadoop Metrics](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Metrics.html).

Metrics collection also requires the OpenTelemetry Java JMX metrics JAR on the collector host. It defaults to `/opt/opentelemetry-java-contrib-jmx-metrics.jar`; adjust the JMX Metrics Collection Jar Path if it lives elsewhere.

For logs, the Hadoop daemons must be writing their log files and the collector must run where it can read those paths. The defaults assume a standard `/usr/local/hadoop/logs/` layout:

* DataNode: `/usr/local/hadoop/logs/hadoop-*-datanode-*.log`
* Resource Manager: `/usr/local/hadoop/logs/hadoop-*-resourcemgr-*.log`
* NameNode: `/usr/local/hadoop/logs/hadoop-*-namenode-*.log`

Adjust the file paths if your installation logs elsewhere.

### Configuration

<figure><img src="/files/b3n4A4reXQHU8esPYWM2" alt="Bindplane docs - Apache Hadoop - image 1"><figcaption></figcaption></figure>

**General**

| Parameter             | Type               | Required | Default               | Description                                                      |
| --------------------- | ------------------ | -------- | --------------------- | ---------------------------------------------------------------- |
| Choose Telemetry Type | Telemetry Selector | No       | `["Logs", "Metrics"]` | Which signals this source collects. Valid values: Logs, Metrics. |

**Logs**

These parameters apply when Logs is selected.

| Parameter                     | Type    | Required | Default                                             | Description                                                                                   |
| ----------------------------- | ------- | -------- | --------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| DataNode Logs                 | Boolean | No       | `true`                                              | Enable to collect DataNode logs.                                                              |
| DataNode File Path(s)         | Strings | No       | `/usr/local/hadoop/logs/hadoop-*-datanode-*.log`    | File paths to tail for DataNode logs. Relevant when DataNode Logs is enabled.                 |
| Resource Manager Logs         | Boolean | No       | `true`                                              | Enable to collect Resource Manager logs.                                                      |
| Resource Manager File Path(s) | Strings | No       | `/usr/local/hadoop/logs/hadoop-*-resourcemgr-*.log` | File paths to tail for Resource Manager logs. Relevant when Resource Manager Logs is enabled. |
| NameNode Logs                 | Boolean | No       | `true`                                              | Enable to collect NameNode logs.                                                              |
| NameNode File Path(s)         | Strings | No       | `/usr/local/hadoop/logs/hadoop-*-namenode-*.log`    | File paths to tail for NameNode logs. Relevant when NameNode Logs is enabled.                 |

**Metrics**

These parameters apply when Metrics is selected.

| Parameter | Type    | Required | Default     | Description                                          |
| --------- | ------- | -------- | ----------- | ---------------------------------------------------- |
| Address   | String  | No       | `localhost` | IP address or hostname to scrape for Hadoop metrics. |
| Port      | Integer | No       | `8004`      | Port to scrape for Hadoop metrics.                   |

**Advanced**

| Parameter                       | Type                 | Required | Default                                           | Description                                                                   |
| ------------------------------- | -------------------- | -------- | ------------------------------------------------- | ----------------------------------------------------------------------------- |
| Start At                        | Enum: beginning, end | No       | `end`                                             | Start reading logs from `beginning` or `end`. Relevant when Logs is selected. |
| Parse                           | Boolean              | No       | `true`                                            | Parses the log fields into structured data.                                   |
| JMX Metrics Collection Jar Path | String               | No       | `/opt/opentelemetry-java-contrib-jmx-metrics.jar` | Full path to the JMX metrics jar. Relevant when Metrics is selected.          |
| Collection Interval             | Integer              | No       | `60`                                              | How often (seconds) to scrape for metrics. Relevant when Metrics is selected. |

### Examples

#### Collect metrics from a NameNode over JMX

This source collects only metrics, scraping the NameNode JMX endpoint on a remote host every 30 seconds.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: hadoop
spec:
  type: hadoop
  parameters:
    - name: telemetry_types
      value:
        - Metrics
    - name: address
      value: 10.0.0.21
    - name: port
      value: 8004
    - name: collection_interval
      value: 30
```

### Configuration Tips

* If the daemon's JVM requires JMX authentication or TLS, make sure those settings are configured on the daemon side and that the collector host can complete the handshake; the JMX endpoint must be reachable from the collector.
* Lower the Collection Interval for finer-grained metrics, but be mindful of the extra load a frequent JMX scrape places on the daemon.
* Each Hadoop daemon (NameNode, DataNode, Resource Manager) exposes its own JMX endpoint on its own port. To monitor multiple daemons, add a separate source instance per daemon address and port.

### Troubleshooting

#### No metrics appear

Symptoms: the source is configured for Metrics but no Hadoop metrics arrive.

Solutions:

1. Confirm JMX remote is enabled on the target daemon's JVM and that the configured Address and Port match its JMX endpoint.
2. Verify the collector host can reach that port (firewall, security group, bind address).
3. Confirm the JMX Metrics Collection Jar Path points to an existing `opentelemetry-java-contrib-jmx-metrics.jar` on the collector host.

#### Connection refused or JMX errors

Symptoms: connection refused, timeouts, or TLS handshake failures when scraping.

Solutions:

1. Check that the daemon is listening on the expected JMX/RMI port and is not bound only to loopback.
2. If the JVM requires authentication or TLS, ensure those are correctly configured on both ends.

#### No logs collected

Symptoms: Logs is enabled but no log records appear.

Solutions:

1. Verify the configured file paths match where the daemons actually write logs; the defaults assume `/usr/local/hadoop/logs/`.
2. Confirm the collector process has read permission on those files.
3. With Start At set to `end`, only new lines written after the source starts are collected; set it to `beginning` to read existing content.

### Standalone Source

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: hadoop
spec:
  type: hadoop
  parameters:
    - name: telemetry_types
      value:
        - Logs
        - Metrics
    - name: address
      value: localhost
    - name: port
      value: 8004
    - name: collection_interval
      value: 60
    - name: enable_namenode_logs
      value: true
    - name: namenode_log_path
      value:
        - /usr/local/hadoop/logs/hadoop-*-namenode-*.log
    - name: enable_datanode_logs
      value: true
    - name: datanode_log_path
      value:
        - /usr/local/hadoop/logs/hadoop-*-datanode-*.log
    - name: enable_resourcemgr_logs
      value: true
    - name: resourcemgr_log_path
      value:
        - /usr/local/hadoop/logs/hadoop-*-resourcemgr-*.log
    - name: start_at
      value: end
    - name: parse
      value: true
```

### Related Resources

* [Apache Hadoop Metrics](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Metrics.html)
* [OpenTelemetry JMX Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jmxreceiver)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.bindplane.com/integrations/sources/hadoop.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.