Azure Event Hub

Prerequisites

The source must have access to an Azure Event Hub with the necessary permissions, the minimum permission being Listen. You can configure resources' Diagnostic settings to send logs to the Azure Event Hub. Read more here: Azure Event Hub.

Each event hub should only accept one telemetry type.

To use Distributed Consumption, you also need an Azure Blob Storage account and an existing container that all participating collectors can read and write to.

Supported Platforms

Bindplane Collector: v1.39.0+ (basic configuration)

Bindplane Collector: v1.97.0+ (distributed consumption, polling controls)

Platform	Metrics	Logs	Traces
Linux	✓	✓
Windows	✓	✓
macOS	✓	✓
Kubernetes Cluster	✓	✓
OpenShift 4 Cluster	✓	✓

Configuration

Field	Description
Telemetry Type	The type of telemetry to gather with this receiver. Only one telemetry type should be sent to a given event hub.
Connection	A string describing the connection to an Azure event hub.
Consumer Group	The Consumer Group to read from. Defaults to $$Default. Dollar signs must be escaped with another dollar sign.
Partition	The partition to watch. If empty, it will watch all partitions. Ignored when Distributed Consumption is enabled.
Offset	The offset at which to start watching the event hub. If empty, starts with the latest offset. If -1, starts at the beginning. Ignored when Distributed Consumption is enabled.
Log Format	The log format to use when parsing logs from Event Hub. Must be one of azure or raw. Raw logs are byte encoded, see the "Raw Encoding" option.
Raw Encoding	The encoding used when decoding the raw logs into human readable text. Defaults to utf-8. The raw byte encoding can be preserved by selecting byte.

Advanced Configuration

Field	Description
Max Poll Events	Maximum number of events to retrieve per poll. Increase to improve throughput. Defaults to 100.
Poll Rate (seconds)	Maximum seconds to wait before returning fewer than Max Poll Events. Lower values reduce latency; higher values can improve batch efficiency. Defaults to 5.
Enable Distributed Consumption	Coordinate partition ownership across multiple collector instances using an Azure Blob Storage checkpoint store. Mutually exclusive with Partition and Offset.
Checkpoint Store Connection String	Connection string for the Azure Blob Storage account used for checkpoint coordination. Required when Distributed Consumption is enabled.
Checkpoint Store Container Name	Blob container used for checkpoint data. The container must already exist before starting the collector. Required when Distributed Consumption is enabled.

Example Configuration

Basic Configuration

For basic configuration, the connection parameter is required. Optionally, specify consumer group, partition, and offset.

Web Interface

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: azure_event_hub
  name: azure_event_hub
spec:
  type: azure_event_hub
  parameters:
    - name: telemetry_types
      value: Logs
    - name: connection
      value: '<your_connection_string>'
    - name: group
      value: '$$Default' # optional, defaults to $$Default
    - name: partition
      value: '' # optional, watches all partitions when empty
    - name: offset
      value: '' # optional, starts at latest when empty
    - name: format
      value: raw # azure or raw
    - name: raw_encoding
      value: utf-8 # only applies when format is raw

Distributed Consumption

When running multiple collector instances against the same event hub, enable Distributed Consumption to coordinate partition ownership through an Azure Blob Storage container. Each partition is leased to a single collector at a time, and ownership rebalances automatically as collectors are added or removed. Progress is checkpointed to the same container, so collectors resume from where they left off after a restart.

The blob container must already exist before starting the collector, and every participating collector must use the same Consumer Group and container. Distributed Consumption is mutually exclusive with the Partition and Offset fields.

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: azure_event_hub
  name: azure_event_hub
spec:
  type: azure_event_hub
  parameters:
    - name: telemetry_types
      value: Logs
    - name: connection
      value: '<your_connection_string>'
    - name: group
      value: '$$Default'
    - name: enable_distributed_consumption
      value: true
    - name: blob_checkpoint_store_connection
      value: '<your_blob_storage_connection_string>'
    - name: blob_checkpoint_store_container_name
      value: 'eventhub-checkpoints'

Tuning Throughput

Use Max Poll Events and Poll Rate to balance latency against batch efficiency. Higher Max Poll Events with a longer Poll Rate yields larger batches, while lower values reduce per-event latency.

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: azure_event_hub
  name: azure_event_hub
spec:
  type: azure_event_hub
  parameters:
    - name: telemetry_types
      value: Logs
    - name: connection
      value: '<your_connection_string>'
    - name: max_poll_events
      value: 500
    - name: poll_rate
      value: 2