Hadoop

The Apache Hadoop source collects metrics over JMX from Hadoop daemons (NameNode, DataNode, Resource Manager) and tails their on-disk log files. Metrics are scraped through the OpenTelemetry Java JMX metrics gatherer using the hadoop target system; logs are read from the daemon log paths and optionally parsed into structured fields.

Supported Telemetry Types

Platform	Metrics	Logs	Traces
macOS	✓	✓
Linux	✓	✓
Windows	✓	✓

Prerequisites

For metrics, the Hadoop daemon you target must have JMX remote enabled and a reachable JMX/RMI port (the source defaults to port 8004). The collector host needs network reachability to that address and port. JVM JMX remote access is typically enabled by setting com.sun.management.jmxremote* system properties (and authentication/TLS where required) in the daemon's HADOOP_*_OPTS environment variables. Metrics are also published through the Metrics2 framework as standard JMX MBeans. See the official Apache Hadoop metrics documentation for details: Apache Hadoop Metrics.

Metrics collection also requires the OpenTelemetry Java JMX metrics JAR on the collector host. It defaults to /opt/opentelemetry-java-contrib-jmx-metrics.jar; adjust the JMX Metrics Collection Jar Path if it lives elsewhere.

For logs, the Hadoop daemons must be writing their log files and the collector must run where it can read those paths. The defaults assume a standard /usr/local/hadoop/logs/ layout:

• DataNode: /usr/local/hadoop/logs/hadoop-*-datanode-*.log
• Resource Manager: /usr/local/hadoop/logs/hadoop-*-resourcemgr-*.log
• NameNode: /usr/local/hadoop/logs/hadoop-*-namenode-*.log

Adjust the file paths if your installation logs elsewhere.

Configuration

General

Parameter	Type	Required	Default	Description
Choose Telemetry Type	Telemetry Selector	No	["Logs", "Metrics"]	Which signals this source collects. Valid values: Logs, Metrics.

Logs

These parameters apply when Logs is selected.

Parameter	Type	Required	Default	Description
DataNode Logs	Boolean	No	true	Enable to collect DataNode logs.
DataNode File Path(s)	Strings	No	/usr/local/hadoop/logs/hadoop-*-datanode-*.log	File paths to tail for DataNode logs. Relevant when DataNode Logs is enabled.
Resource Manager Logs	Boolean	No	true	Enable to collect Resource Manager logs.
Resource Manager File Path(s)	Strings	No	/usr/local/hadoop/logs/hadoop-*-resourcemgr-*.log	File paths to tail for Resource Manager logs. Relevant when Resource Manager Logs is enabled.
NameNode Logs	Boolean	No	true	Enable to collect NameNode logs.
NameNode File Path(s)	Strings	No	/usr/local/hadoop/logs/hadoop-*-namenode-*.log	File paths to tail for NameNode logs. Relevant when NameNode Logs is enabled.

Metrics

These parameters apply when Metrics is selected.

Parameter	Type	Required	Default	Description
Address	String	No	localhost	IP address or hostname to scrape for Hadoop metrics.
Port	Integer	No	8004	Port to scrape for Hadoop metrics.

Advanced

Parameter	Type	Required	Default	Description
Start At	Enum: beginning, end	No	end	Start reading logs from beginning or end. Relevant when Logs is selected.
Parse	Boolean	No	true	Parses the log fields into structured data.
JMX Metrics Collection Jar Path	String	No	/opt/opentelemetry-java-contrib-jmx-metrics.jar	Full path to the JMX metrics jar. Relevant when Metrics is selected.
Collection Interval	Integer	No	60	How often (seconds) to scrape for metrics. Relevant when Metrics is selected.

Examples

Collect metrics from a NameNode over JMX

This source collects only metrics, scraping the NameNode JMX endpoint on a remote host every 30 seconds.

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: hadoop
spec:
  type: hadoop
  parameters:
    - name: telemetry_types
      value:
        - Metrics
    - name: address
      value: 10.0.0.21
    - name: port
      value: 8004
    - name: collection_interval
      value: 30

Configuration Tips

• If the daemon's JVM requires JMX authentication or TLS, make sure those settings are configured on the daemon side and that the collector host can complete the handshake; the JMX endpoint must be reachable from the collector.
• Lower the Collection Interval for finer-grained metrics, but be mindful of the extra load a frequent JMX scrape places on the daemon.
• Each Hadoop daemon (NameNode, DataNode, Resource Manager) exposes its own JMX endpoint on its own port. To monitor multiple daemons, add a separate source instance per daemon address and port.

Troubleshooting

No metrics appear

Symptoms: the source is configured for Metrics but no Hadoop metrics arrive.

Solutions:

1. Confirm JMX remote is enabled on the target daemon's JVM and that the configured Address and Port match its JMX endpoint.
2. Verify the collector host can reach that port (firewall, security group, bind address).
3. Confirm the JMX Metrics Collection Jar Path points to an existing opentelemetry-java-contrib-jmx-metrics.jar on the collector host.

Connection refused or JMX errors

Symptoms: connection refused, timeouts, or TLS handshake failures when scraping.

Solutions:

1. Check that the daemon is listening on the expected JMX/RMI port and is not bound only to loopback.
2. If the JVM requires authentication or TLS, ensure those are correctly configured on both ends.

No logs collected

Symptoms: Logs is enabled but no log records appear.

Solutions:

1. Verify the configured file paths match where the daemons actually write logs; the defaults assume /usr/local/hadoop/logs/.
2. Confirm the collector process has read permission on those files.
3. With Start At set to end, only new lines written after the source starts are collected; set it to beginning to read existing content.

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  name: hadoop
spec:
  type: hadoop
  parameters:
    - name: telemetry_types
      value:
        - Logs
        - Metrics
    - name: address
      value: localhost
    - name: port
      value: 8004
    - name: collection_interval
      value: 60
    - name: enable_namenode_logs
      value: true
    - name: namenode_log_path
      value:
        - /usr/local/hadoop/logs/hadoop-*-namenode-*.log
    - name: enable_datanode_logs
      value: true
    - name: datanode_log_path
      value:
        - /usr/local/hadoop/logs/hadoop-*-datanode-*.log
    - name: enable_resourcemgr_logs
      value: true
    - name: resourcemgr_log_path
      value:
        - /usr/local/hadoop/logs/hadoop-*-resourcemgr-*.log
    - name: start_at
      value: end
    - name: parse
      value: true

Related Resources

• Apache Hadoop Metrics
• OpenTelemetry JMX Receiver