Azure Event Hub

The Azure Event Hub source reads telemetry from an Azure Event Hub and pushes it through the collector pipeline. Azure resources commonly stream their diagnostic logs and platform metrics to an event hub through diagnostic settings, which this source consumes. Each event hub should carry only one telemetry type.

Supported Telemetry Types

Platform	Metrics	Logs	Traces
Linux	✓	✓
Windows	✓	✓
macOS	✓	✓
Kubernetes Cluster	✓	✓
OpenShift 4 Cluster	✓	✓

Prerequisites

• An Azure Event Hub the collector can reach, with at least the Listen permission.
• Either a connection string for the event hub, or an Azure identity (managed identity, workload identity, or service principal) with Listen on the event hub.
• Source resources configured to stream to the event hub, for example through diagnostic settings.
• For Distributed Consumption: an Azure Blob Storage account and an existing container that all participating collectors can read and write.

Configuration

General

Parameter	Type	Default	Description
Choose Telemetry Type	Telemetry Selector	Logs	Type of telemetry to accept (Logs or Metrics). Only one telemetry type should be sent to a given event hub.
Consumer Group	String	$$Default	The Consumer Group to read from. Defaults to $$Default if empty. Dollar signs must be escaped with another dollar sign.
Partition	String		The partition to watch. If empty, all partitions are watched. Ignored when Distributed Consumption is enabled.
Offset	String		The offset at which to start watching the event hub. If empty, starts with the latest offset. If -1, starts at the beginning. Ignored when Distributed Consumption is enabled.

Authentication

By default the source authenticates with a connection string. Enable Use Azure Identity Authentication (enable_external_auth, default false) to authenticate with an Azure identity instead.

Parameter	Type	Default	Description
Connection String	String		Connection string describing the connection to the event hub. Required when Azure identity authentication is disabled. Sensitive.
Event Hub Name	String		Name of the event hub to read from. Required when Azure identity authentication is enabled.
Event Hub Namespace	String		Fully qualified Event Hubs namespace, e.g. my-namespace.servicebus.windows.net. Required when Azure identity authentication is enabled.
Authentication Type	Extension	azure_auth	The Azure identity method used to authenticate (managed identity, workload identity, or service principal). Required when Azure identity authentication is enabled.

Log Format

Applies only when telemetry_types is Logs.

Parameter	Type	Default	Description
Log Format	Enum	azure	Format of the logs sent to the event hub. One of azure or raw. Raw logs are byte encoded. Required.
Raw Encoding	Enum	utf-8	Encoding of raw logs. One of utf-8, utf-16, nop, or byte. When set to byte, decoding is disabled. Applies only when format is raw.

Advanced

Parameter	Type	Default	Description
Max Poll Events	Integer	100	Maximum number of events to retrieve per poll. Increase to improve throughput. Minimum 1.
Poll Rate (seconds)	Integer	5	Maximum seconds to wait before returning fewer than Max Poll Events. Lower values reduce latency; higher values can improve batch efficiency. Minimum 1.
Enable Prefetching	Boolean	true	Prefetch events from the Event Hub. Improves throughput at the cost of additional memory usage.
Prefetch Count	Integer	300	Number of events to prefetch. Increase to improve throughput at the cost of additional memory. Applies only when prefetching is enabled. Minimum 1.
Enable Distributed Consumption	Boolean	false	Coordinate partition ownership across multiple collector instances using an Azure Blob Storage checkpoint store. Mutually exclusive with partition and offset.
Checkpoint Store Connection String	String		Connection string for the Blob Storage account used for checkpoint coordination. Required when Distributed Consumption is enabled and Azure identity authentication is disabled. Sensitive.
Checkpoint Store Storage Account URL	String		Blob service URL of the storage account used for checkpoint coordination, e.g. https://myaccount.blob.core.windows.net. Required when Distributed Consumption and Azure identity authentication are both enabled. The Azure identity must have access to the container.
Checkpoint Store Container Name	String		Blob container used for checkpoint data. The container must already exist before starting the collector. Required when Distributed Consumption is enabled.

Example Configuration

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: azure_event_hub
  name: azure_event_hub
spec:
  type: azure_event_hub
  parameters:
    - name: telemetry_types
      value: 'Logs'
    - name: connection
      value: '<your_connection_string>'
    - name: group
      value: '$$Default'
    - name: format
      value: 'azure'

Configuration Tips

Authenticating with an Azure identity

• Set enable_external_auth to true, then provide event_hub_name and event_hub_namespace and choose an authentication method through auth_extension. The identity needs at least Listen on the event hub, plus read/write access to the checkpoint container when Distributed Consumption is enabled.
• Azure identity authentication uses the Azure Auth extension. Managed identity, workload identity, and service principal are all supported.

Distributed Consumption

• When running multiple collectors against the same event hub, enable enable_distributed_consumption to coordinate partition ownership through an Azure Blob Storage container. Each partition is leased to a single collector at a time and ownership rebalances automatically as collectors are added or removed.
• The container must already exist before the collector starts, and every participating collector must use the same group and container.
• Distributed Consumption is mutually exclusive with partition and offset. Leave those unset when it is enabled.

Tuning throughput

• Use max_poll_events and poll_rate to balance latency against batch efficiency. Larger max_poll_events and poll_rate increase batch size and throughput but add event latency; smaller values reduce per-event latency at the cost of smaller batches.
• When enable_prefetch is on (the default), the receiver buffers events before processing. Raising prefetch_count (default 300) can improve throughput under heavy load but increases memory consumption.

Troubleshooting

Symptom: The collector starts but no logs or metrics arrive. Solution: Confirm the credentials grant at least the Listen permission on the event hub, and that the source resources are actually streaming to it (for example, that diagnostic settings target the correct event hub). Verify group matches an existing Consumer Group.

Symptom: Records arrive but the telemetry type is wrong, or telemetry is dropped. Solution: Send only one telemetry type to a given event hub and set telemetry_types to match. A logs source pointed at a metrics event hub (or the reverse) will not parse the data correctly.

Symptom: Raw logs appear garbled or unreadable. Solution: Set raw_encoding to the encoding the producer uses (utf-8, utf-16, or nop). Use byte only when you intend to preserve the raw byte encoding without decoding.

Symptom: Multiple collectors reprocess the same events, or partitions are not balanced across instances. Solution: Enable enable_distributed_consumption and point every collector at the same Blob Storage container and group. Ensure the container exists ahead of time and that the credentials (connection string or Azure identity) can read and write it. Do not set partition or offset while Distributed Consumption is enabled.

Related Resources

• Azure Event Hubs documentation
• Get an Event Hubs connection string
• Azure Monitor diagnostic settings
• Azure Blob Storage overview