> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/sources/kubernetes-gateway-azure-event-hub.md).

# Kubernetes Gateway Azure Event Hub

Consume Azure Event Hub logs and metrics across a horizontally scaled Kubernetes Gateway. Distributed Consumption is always enabled so replicas share partitions through a Blob Storage checkpoint store

The Kubernetes Gateway Azure Event Hub source is the Gateway variant of the [Azure Event Hub source](/integrations/sources/azure-event-hub.md). Gateway collectors run as multiple replicas, so this source requires Distributed Consumption to be enabled and coordinates partition ownership through an [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) checkpoint store. Each replica leases a subset of partitions, and ownership rebalances as replicas are added or removed. For single-collector deployments, use the standard [Azure Event Hub source](/integrations/sources/azure-event-hub.md) instead.

### Supported Telemetry Types

| Platform            | Metrics | Logs | Traces |
| ------------------- | ------- | ---- | ------ |
| Kubernetes Gateway  | ✓       | ✓    |        |
| OpenShift 4 Gateway | ✓       | ✓    |        |

### Prerequisites

* An Azure Event Hub the collectors can reach, with at least the `Listen` permission.
* An Azure Blob Storage account and an existing container for checkpoint data. The container must exist before the collectors start, and every replica must be able to read, write, and list blobs in it.
* Either connection strings for the event hub and storage account, or one [Azure identity](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview) (managed identity, workload identity, or service principal) with access to both.
* Source resources configured to stream to the event hub, for example through [diagnostic settings](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings).

#### Required Permissions

| Resource                          | Connection String                                                          | Azure Identity                                                           |
| --------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| Event Hub                         | Shared access policy with `Listen`                                         | `Azure Event Hubs Data Receiver` role on the event hub or namespace      |
| Blob Storage checkpoint container | Account key or SAS with read, write, and list permissions on the container | `Storage Blob Data Contributor` role on the container or storage account |

When Azure identity authentication is enabled, the same identity is used for both the event hub and the checkpoint store, so it needs both roles.

### Configuration

<figure><img src="https://1405008107-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgmiOMzBfoNFwmKJFHMcJ%2Fuploads%2FK0ZjzIVqUH8T5NkCO9u8%2Fimage.png?alt=media&amp;token=fd77890c-e550-442b-97f6-0ae47b69a2a2" alt=""><figcaption></figcaption></figure>

#### General

| Parameter             | Type               | Default     | Description                                                                                                               |
| --------------------- | ------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------- |
| Choose Telemetry Type | Telemetry Selector | `Logs`      | Type of telemetry to accept (`Logs` or `Metrics`). Only one telemetry type should be sent to a given event hub.           |
| Consumer Group        | String             | `$$Default` | The Consumer Group to read from. Defaults to `$$Default` if empty. Dollar signs must be escaped with another dollar sign. |

The `Partition` and `Offset` parameters from the standard source are not available. Distributed Consumption assigns partitions and resumes from the checkpointed offset.

#### Authentication

**Authentication Method** (`auth_extension`) defaults to **Connection String**. Select an Azure Auth extension instead to authenticate with an Azure identity (managed identity, workload identity, or service principal). The same method is used for both the event hub and the checkpoint store.

| Parameter             | Type      | Default             | Description                                                                                                                              |
| --------------------- | --------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Authentication Method | Extension | `connection_string` | **Connection String**, or an Azure Auth extension for identity-based authentication. Required.                                           |
| Connection String     | String    |                     | Connection string describing the connection to the event hub. Required when the authentication method is `connection_string`. Sensitive. |
| Event Hub Name        | String    |                     | Name of the event hub to read from. Required when using an Azure identity.                                                               |
| Event Hub Namespace   | String    |                     | Fully qualified Event Hubs namespace, e.g. `my-namespace.servicebus.windows.net`. Required when using an Azure identity.                 |

#### Checkpoint Store

| Parameter                            | Type   | Default | Description                                                                                                                                                 |
| ------------------------------------ | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Checkpoint Store Connection String   | String |         | Connection string for the Blob Storage account used for checkpoint coordination. Required when the authentication method is `connection_string`. Sensitive. |
| Checkpoint Store Storage Account URL | String |         | Blob service URL of the storage account, e.g. `https://myaccount.blob.core.windows.net`. Required when using an Azure identity.                             |
| Checkpoint Store Container Name      | String |         | Blob container used for checkpoint data. The container must already exist before starting the collectors. Required.                                         |

#### Log Format

Applies only when `telemetry_types` is `Logs`.

| Parameter    | Type | Default | Description                                                                                                                                      |
| ------------ | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| Log Format   | Enum | `azure` | Format of the logs sent to the event hub. One of `azure` or `raw`. Raw logs are byte encoded. Required.                                          |
| Raw Encoding | Enum | `utf-8` | Encoding of raw logs. One of `utf-8`, `utf-16`, `nop`, or `byte`. When set to `byte`, decoding is disabled. Applies only when `format` is `raw`. |

#### Advanced

| Parameter           | Type    | Default | Description                                                                                                                                                |
| ------------------- | ------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Max Poll Events     | Integer | `100`   | Maximum number of events to retrieve per poll. Increase to improve throughput. Minimum `1`.                                                                |
| Poll Rate (seconds) | Integer | `5`     | Maximum seconds to wait before returning fewer than Max Poll Events. Lower values reduce latency; higher values can improve batch efficiency. Minimum `1`. |
| Enable Prefetching  | Boolean | `true`  | Prefetch events from the Event Hub. Improves throughput at the cost of additional memory usage.                                                            |
| Prefetch Count      | Integer | `300`   | Number of events to prefetch. Increase to improve throughput at the cost of additional memory. Applies only when prefetching is enabled. Minimum `1`.      |

### Example Configuration

#### Connection Strings

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: k8s_gateway_azure_event_hub
  name: k8s_gateway_azure_event_hub
spec:
  type: k8s_gateway_azure_event_hub
  parameters:
    - name: telemetry_types
      value: 'Logs'
    - name: connection
      value: '<your_event_hub_connection_string>'
    - name: group
      value: '$$Default'
    - name: blob_checkpoint_store_connection
      value: '<your_blob_storage_connection_string>'
    - name: blob_checkpoint_store_container_name
      value: 'eventhub-checkpoints'
    - name: format
      value: 'azure'
```

#### Azure Workload Identity

Workload identity is the typical choice on AKS. The identity needs `Azure Event Hubs Data Receiver` on the event hub and `Storage Blob Data Contributor` on the checkpoint container.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: k8s_gateway_azure_event_hub
  name: k8s_gateway_azure_event_hub
spec:
  type: k8s_gateway_azure_event_hub
  parameters:
    - name: telemetry_types
      value: 'Logs'
    - name: event_hub_name
      value: 'my-hub'
    - name: event_hub_namespace
      value: 'my-namespace.servicebus.windows.net'
    - name: auth_extension
      value:
        type: azure_auth
        parameters:
          - name: auth_method
            value: workload_identity
          - name: workload_identity_client_id
            value: '<application_client_id>'
          - name: workload_identity_tenant_id
            value: '<tenant_id>'
          - name: workload_identity_token_file
            value: '/var/run/secrets/azure/tokens/azure-identity-token'
    - name: group
      value: '$$Default'
    - name: blob_checkpoint_store_storage_account_url
      value: 'https://myaccount.blob.core.windows.net'
    - name: blob_checkpoint_store_container_name
      value: 'eventhub-checkpoints'
```

### Configuration Tips

#### How scaling works

* Each partition is leased to exactly one Gateway replica at a time. When replicas are added or removed, ownership rebalances automatically.
* Progress is checkpointed to the blob container, so a replica that restarts or takes over a partition resumes from the last checkpoint rather than the latest offset.
* Running more replicas than the event hub has partitions leaves the extra replicas idle for this source. Size the Gateway to the partition count, or increase partitions on the event hub.
* All replicas share the same configuration, so they automatically use the same Consumer Group and checkpoint container. Use a separate container or Consumer Group for each event hub.

#### Authenticating with an Azure identity

* Set `auth_extension` to an Azure Auth extension, then provide `event_hub_name`, `event_hub_namespace`, and `blob_checkpoint_store_storage_account_url`. The `connection` and `blob_checkpoint_store_connection` parameters are ignored.
* Azure identity authentication uses the [Azure Auth extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/azureauthextension). Managed identity, workload identity, and service principal are all supported. See Authenticating with an Azure identity on the standard source for details.

### Troubleshooting

**Collectors start, but no logs or metrics arrive**

* Confirm the credentials grant at least `Listen` on the event hub.
* Confirm `group` matches an existing Consumer Group.
* Verify the source resources are actually streaming to the event hub.

**Collectors fail to start, or log errors about the checkpoint container**

* Create the container before starting the collectors.
* Confirm the storage credentials can read, write, and list blobs in it.
* When Azure identity is enabled, the identity needs `Storage Blob Data Contributor`, reader access isn't enough.

**Some replicas receive no events**

* Compare the replica count to the event hub partition count. Replicas beyond the partition count have nothing to lease.

**Multiple replicas reprocess the same events**

* Confirm every replica points at the same storage account and container.
* Confirm no other source or collector uses the same `Consumer Group` and container against a different event hub.

### Related Resources

* [Azure Event Hub source](/integrations/sources/azure-event-hub.md)
* [Azure Event Hubs documentation](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about)
* [Balance partition load across multiple instances](https://learn.microsoft.com/en-us/azure/event-hubs/event-processor-balance-partition-load)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.bindplane.com/integrations/sources/kubernetes-gateway-azure-event-hub.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.