Badger

Overview

Badger is a high-performance key-value storage engine optimized for maximum throughput. It trades increased memory usage for significantly better per-collector performance, making it ideal for high throughput deployments.

Under the hood, Badger Persistent Queue storage is the Bindplane badger extension.

Configuration

The following parameters control Badger storage behavior:

Parameter	Type	Default	Description
Persistent Queue Storage	string	${OIQ_OTEL_COLLECTOR_HOME}/storage	Directory where Badger stores data files
Synchronize Persistent Queue to Disk	bool	true	Whether to sync writes immediately to disk for durability
Memory Table Size	int (bytes)	64 MiB	Size of each memtable in bytes. Larger values improve write performance but use more memory
Memory Block Cache Size	int (bytes)	256 MiB	Size of block cache in bytes. Larger values improve read performance but use more memory.
Number of Compactors	int	12	Number of compaction workers to use.
Number of Level Zero Tables	int	2	Number of level zero tables to keep in memory.
Number of Level Zero Tables Stall	int	0	Number of level zero tables that stalls writes.
Memory Value Log File Size	int (bytes)	64 MiB	Maximum size of each value log file in bytes. Smaller values reduce baseline disk usage but may increase file count.
Blob Garbage Collection Interval	int (seconds)	5m	Interval at which garbage collection on is run to cleanup the disk space used to store telemetry to be sent.
Blob Garbage Collection Discard Ratio	float	0.5	Fraction of invalid data in a value log file to trigger garbage collection.

When to Use Badger

Use Badger for:

• Large-scale gateways (5+ collectors)

• Deployments requiring maximum per-collector throughput

• Multi-region or multi-zone architectures

• When infrastructure where per-collector efficiency justifies higher memory use

Badger provides higher throughput per collector, reducing the total number of collectors needed at large scales.

When Pebble is Better

Consider using Pebble if:

• Standard workflows that don't value minimal performance impact from the queue

• Don't need maximum per-collector throughput

See Pebble for standard deployment guidance.

Monitoring

Monitor these indicators after configuring Badger:

Disk Usage

• Growth is normal and related to data volume

• Badger handles compaction automatically in background

• Ensure sufficient disk space remains available

Per-Collector Throughput

• Should be significantly higher than Pebble

• Use to determine when to add additional collectors

• Indicates if efficiency gains are being realized

Memory Usage

• Higher than Pebble (intentional trade-off for throughput)

• Should be predictable based on configuration

Troubleshooting

High disk usage:

• Normal with high-throughput deployments

• Badger compaction happens automatically in background

• May take time to reclaim space from deletions

• Verify adequate disk space remains available

Memory usage higher than expected:

• This is the intended trade-off for better throughput

• If critical, reduce Block Cache Size or Table Size

• Consider Pebble if memory is severely constrained

Per-collector throughput not improved:

• Check if CPU or disk I/O is saturated

• May need infrastructure-level optimization

Scaling Considerations

Badger's higher memory usage:

• Infrastructure efficiency gains from fewer collectors exceed per-collector memory cost

• Each collector can handle more throughput, deferring scale-out

• Better resource utilization at large scales

Scaling efficiency:

• Badger's higher memory cost per collector is offset by needing fewer total collectors

• This becomes advantageous at collector scale where infrastructure costs are significant

Next Steps

1. Follow the Switching Queues guide to avoid data loss

2. Monitor per-collector throughput, memory usage, and disk space

3. Adjust compaction settings if disk usage remains high

For help choosing a persistent queue storage backend, see Persistent Queue.