Kubernetes Gateway Azure Event Hub

Consume Azure Event Hub logs and metrics across a horizontally scaled Kubernetes Gateway. Distributed Consumption is always enabled so replicas share partitions through a Blob Storage checkpoint store

The Kubernetes Gateway Azure Event Hub source is the Gateway variant of the Azure Event Hub source. Gateway collectors run as multiple replicas, so this source requires Distributed Consumption to be enabled and coordinates partition ownership through an Azure Blob Storage checkpoint store. Each replica leases a subset of partitions, and ownership rebalances as replicas are added or removed. For single-collector deployments, use the standard Azure Event Hub source instead.

Supported Telemetry Types

Platform	Metrics	Logs	Traces
Kubernetes Gateway	✓	✓
OpenShift 4 Gateway	✓	✓

Prerequisites

• An Azure Event Hub the collectors can reach, with at least the Listen permission.

• An Azure Blob Storage account and an existing container for checkpoint data. The container must exist before the collectors start, and every replica must be able to read, write, and list blobs in it.

• Either connection strings for the event hub and storage account, or one Azure identity (managed identity, workload identity, or service principal) with access to both.

• Source resources configured to stream to the event hub, for example through diagnostic settings.

Required Permissions

Resource	Connection String	Azure Identity
Event Hub	Shared access policy with Listen	Azure Event Hubs Data Receiver role on the event hub or namespace
Blob Storage checkpoint container	Account key or SAS with read, write, and list permissions on the container	Storage Blob Data Contributor role on the container or storage account

When Azure identity authentication is enabled, the same identity is used for both the event hub and the checkpoint store, so it needs both roles.

Configuration

General

Parameter	Type	Default	Description
Choose Telemetry Type	Telemetry Selector	Logs	Type of telemetry to accept (Logs or Metrics). Only one telemetry type should be sent to a given event hub.
Consumer Group	String	$$Default	The Consumer Group to read from. Defaults to $$Default if empty. Dollar signs must be escaped with another dollar sign.

The Partition and Offset parameters from the standard source are not available. Distributed Consumption assigns partitions and resumes from the checkpointed offset.

Authentication

Authentication Method (auth_extension) defaults to Connection String. Select an Azure Auth extension instead to authenticate with an Azure identity (managed identity, workload identity, or service principal). The same method is used for both the event hub and the checkpoint store.

Parameter	Type	Default	Description
Authentication Method	Extension	connection_string	Connection String, or an Azure Auth extension for identity-based authentication. Required.
Connection String	String		Connection string describing the connection to the event hub. Required when the authentication method is connection_string. Sensitive.
Event Hub Name	String		Name of the event hub to read from. Required when using an Azure identity.
Event Hub Namespace	String		Fully qualified Event Hubs namespace, e.g. my-namespace.servicebus.windows.net. Required when using an Azure identity.

Checkpoint Store

Parameter	Type	Default	Description
Checkpoint Store Connection String	String		Connection string for the Blob Storage account used for checkpoint coordination. Required when the authentication method is connection_string. Sensitive.
Checkpoint Store Storage Account URL	String		Blob service URL of the storage account, e.g. https://myaccount.blob.core.windows.net. Required when using an Azure identity.
Checkpoint Store Container Name	String		Blob container used for checkpoint data. The container must already exist before starting the collectors. Required.

Log Format

Applies only when telemetry_types is Logs.

Parameter	Type	Default	Description
Log Format	Enum	azure	Format of the logs sent to the event hub. One of azure or raw. Raw logs are byte encoded. Required.
Raw Encoding	Enum	utf-8	Encoding of raw logs. One of utf-8, utf-16, nop, or byte. When set to byte, decoding is disabled. Applies only when format is raw.

Advanced

Parameter	Type	Default	Description
Max Poll Events	Integer	100	Maximum number of events to retrieve per poll. Increase to improve throughput. Minimum 1.
Poll Rate (seconds)	Integer	5	Maximum seconds to wait before returning fewer than Max Poll Events. Lower values reduce latency; higher values can improve batch efficiency. Minimum 1.
Enable Prefetching	Boolean	true	Prefetch events from the Event Hub. Improves throughput at the cost of additional memory usage.
Prefetch Count	Integer	300	Number of events to prefetch. Increase to improve throughput at the cost of additional memory. Applies only when prefetching is enabled. Minimum 1.

Example Configuration

Connection Strings

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: k8s_gateway_azure_event_hub
  name: k8s_gateway_azure_event_hub
spec:
  type: k8s_gateway_azure_event_hub
  parameters:
    - name: telemetry_types
      value: 'Logs'
    - name: connection
      value: '<your_event_hub_connection_string>'
    - name: group
      value: '$$Default'
    - name: blob_checkpoint_store_connection
      value: '<your_blob_storage_connection_string>'
    - name: blob_checkpoint_store_container_name
      value: 'eventhub-checkpoints'
    - name: format
      value: 'azure'

Azure Workload Identity

Workload identity is the typical choice on AKS. The identity needs Azure Event Hubs Data Receiver on the event hub and Storage Blob Data Contributor on the checkpoint container.

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: k8s_gateway_azure_event_hub
  name: k8s_gateway_azure_event_hub
spec:
  type: k8s_gateway_azure_event_hub
  parameters:
    - name: telemetry_types
      value: 'Logs'
    - name: event_hub_name
      value: 'my-hub'
    - name: event_hub_namespace
      value: 'my-namespace.servicebus.windows.net'
    - name: auth_extension
      value:
        type: azure_auth
        parameters:
          - name: auth_method
            value: workload_identity
          - name: workload_identity_client_id
            value: '<application_client_id>'
          - name: workload_identity_tenant_id
            value: '<tenant_id>'
          - name: workload_identity_token_file
            value: '/var/run/secrets/azure/tokens/azure-identity-token'
    - name: group
      value: '$$Default'
    - name: blob_checkpoint_store_storage_account_url
      value: 'https://myaccount.blob.core.windows.net'
    - name: blob_checkpoint_store_container_name
      value: 'eventhub-checkpoints'

Configuration Tips

How scaling works

• Each partition is leased to exactly one Gateway replica at a time. When replicas are added or removed, ownership rebalances automatically.

• Progress is checkpointed to the blob container, so a replica that restarts or takes over a partition resumes from the last checkpoint rather than the latest offset.

• Running more replicas than the event hub has partitions leaves the extra replicas idle for this source. Size the Gateway to the partition count, or increase partitions on the event hub.

• All replicas share the same configuration, so they automatically use the same Consumer Group and checkpoint container. Use a separate container or Consumer Group for each event hub.

Authenticating with an Azure identity

• Set auth_extension to an Azure Auth extension, then provide event_hub_name, event_hub_namespace, and blob_checkpoint_store_storage_account_url. The connection and blob_checkpoint_store_connection parameters are ignored.

• Azure identity authentication uses the Azure Auth extension. Managed identity, workload identity, and service principal are all supported. See Authenticating with an Azure identity on the standard source for details.

Troubleshooting

Collectors start, but no logs or metrics arrive

• Confirm the credentials grant at least Listen on the event hub.

• Confirm group matches an existing Consumer Group.

• Verify the source resources are actually streaming to the event hub.

Collectors fail to start, or log errors about the checkpoint container

• Create the container before starting the collectors.

• Confirm the storage credentials can read, write, and list blobs in it.

• When Azure identity is enabled, the identity needs Storage Blob Data Contributor, reader access isn't enough.

Some replicas receive no events

• Compare the replica count to the event hub partition count. Replicas beyond the partition count have nothing to lease.

Multiple replicas reprocess the same events

• Confirm every replica points at the same storage account and container.

• Confirm no other source or collector uses the same Consumer Group and container against a different event hub.

Related Resources

• Azure Event Hub source

• Azure Event Hubs documentation

• Balance partition load across multiple instances