Formatting Cluster Level Events from Kubernetes for Querying in ClickStack

Kubernetes cluster events contain valuable operational signals about resource state changes, but raw event streams generate noise that obscures meaningful alerts.

Using the Process Kubernetes Cluster Events for Clickhouse blueprint, you can filter routine controller reconciliation, reduce cardinality from object UIDs, map event types to severity levels, and deduplicate warnings, preparing cluster events for efficient Clickhouse storage and alerting.

Overview

The Kubernetes events blueprint preprocesses cluster-level events from the Kubernetes Events API before they reach ClickHouse. It solves specific challenges in Kubernetes event analytics:

• Controller Noise Filtering: Controllers continuously reconcile desired state. Events like ScalingReplicaSet, SuccessfulCreate, SuccessfulDelete, Scheduled, and Pulled generate high volume with low signal. The blueprint filters these routine Normal events.

• Cardinality Explosion Prevention: Kubernetes object UIDs (pod UIDs, deployment UIDs, node UIDs) are globally unique and create massive cardinality in Clickhouse's map columns, degrading query performance. The blueprint removes these identifiers.

• Severity Mapping: Kubernetes events carry type (Normal/Warning) but lack OTLP severity levels. The blueprint maps event types and reasons to SeverityText and SeverityNumber, enabling efficient SQL queries like WHERE SeverityText = 'WARN'.

• Warning Deduplication: Controllers can emit repeated warnings every few seconds during resource pressure or failed rollouts. The blueprint collapses these into single entries with event counts.

Bindplane Configuration

To deploy this blueprint in your Bindplane instance:

1. Navigate to Blueprints and choose the Process Kubernetes Cluster Events for ClickHouse Blueprint. Save it to your Library.

2. Open the Processors section in any of your Bindplane configurations.

3. Modify the Blueprint to fit your specific dataset and requirements.

4. Ensure that the data is formatted correctly by comparing the Snapshot you see to the Live Preview, and validating data pre-processing looks good.

5. Save your changes and rollout the configuration update to production.

Resource Detection Setup

The first processor in the pipeline detects resource attributes based on your cloud environment. Update the detector to match your infrastructure:

• GCP: Set detectors: ["gcp"] (default)

• AWS EKS: Set detectors: ["eks"]

• AWS EC2: Set detectors: ["ec2"]

• Azure AKS: Set detectors: ["aks"]

• Azure: Set detectors: ["azure"]

Select the detector matching your Kubernetes cluster's hosting environment. This ensures Clickhouse stores accurate resource metadata in the ResourceAttributes map column.

Processing Pipeline

The blueprint applies these transformations in sequence:

Resource Enrichment: Detects and adds resource attributes (cloud provider, account, region, availability zone) based on your environment. These attributes populate Clickhouse's ResourceAttributes column for multi-cloud deployments.

Controller Noise Filtering: Normal events from Kubernetes system components are excluded:

• ScalingReplicaSet – ReplicaSet scaling during deployments

• SuccessfulCreate / SuccessfulDelete – Object creation/deletion

• Scheduled – Pod scheduling to nodes

• Started / Created / Pulled – Container lifecycle events

• SawCompletedJob – Job completion

• Killing – Pod termination

• SuccessfulRescale – HPA scaling

These events are routine and add minimal operational value. Warning and Error events are always preserved.

High-Cardinality UID Removal: Kubernetes object UIDs are deleted:

• Pod UIDs, ReplicaSet UIDs, Deployment UIDs

• StatefulSet, DaemonSet, Job, CronJob UIDs

• Node UIDs, Namespace UIDs, Service UIDs

Removing UIDs prevents fragmentation in Clickhouse's LogAttributes and ResourceAttributes map columns.

Metadata Normalization: Standard fields are set if missing:

• service.name defaults to kubernetes

• k8s.cluster.name defaults to default (update to your actual cluster name if desired)

• k8s.namespace.name defaults to default

These defaults ensure consistent Clickhouse schema across all cluster events.

Event Type to Severity Mapping: Kubernetes event types and reasons are mapped to OTLP severity:

• Normal events → SeverityText="INFO", SeverityNumber=9

• Warning events → SeverityText="WARN", SeverityNumber=13

• Events with reasons containing "error", "failed", "backoff", or "crash" → SeverityText="ERROR", SeverityNumber=17

This mapping enables native ClickHouse severity filtering without parsing string fields.

Warning Deduplication: Warning events are deduplicated within 60-second windows. Repeated warnings (e.g., "PodCrashLooping" during cascading failures) are collapsed into single entries with an event_count field, reducing row count in ClickHouse.

Batching: Events are batched for efficient ingestion.

Customizing the Blueprint

Adjust the blueprint for your deployment:

• Preserve Routine Events: Remove or modify the "Filter K8s Controller Noise" processor if you need complete event visibility

• Adjust Cluster Name: Update the default k8s.cluster.name value to match your actual cluster name for easier filtering in ClickHouse

• Change Dedup Window: Modify the warning deduplication interval (currently 60 seconds) to tune sensitivity

• Add Custom Filters: Insert additional processors to filter other high-volume events specific to your environment

Querying Deduplicated Events in ClickHouse

When working with deduplicated events, include the event_count field:

SELECT
    attributes['k8s.event.reason'] as reason,
    attributes['involved_object.name'] as object_name,
    event_count,
    COUNT() as unique_events
FROM otel_logs
WHERE SeverityText = 'WARN'
  AND attributes['k8s.cluster.name'] = 'production'
GROUP BY reason, object_name, event_count
ORDER BY unique_events DESC

This query shows warning events grouped by reason and object, with event_count indicating repetition frequency.

Integration with ClickHouse

Configure your ClickHouse exporter to align with the blueprint:

• Batch size: 5000-10000 rows

• Timeout: 5+ seconds for adequate batching

• Connection pooling: Enabled for throughput

• Retry logic: Enabled for network resilience

The blueprint's filtering and deduplication typically reduce raw event volume by 60-80%, significantly lowering storage costs.

Monitoring Pipeline Effectiveness

Track these metrics to ensure healthy operation:

• Filter Effectiveness: Normal events should represent 40-70% of raw event volume (these are filtered)

• Dedup Rate: Warning events with event_count > 1 indicate successful deduplication

• Cardinality Reduction: UID field removal prevents LogAttributes from fragmenting

• Severity Distribution: Verify that mapped events have consistent SeverityText/SeverityNumber values

NOTE

This Blueprint has been tested against standard data patterns. You may need to adjust the configuration to match your specific data format.