Persistent Queue

Persistent Queue storage buffers telemetry data temporarily, preventing data loss when destinations are unavailable or during collector maintenance.

Badger Persistent Queue – Not Recommended

Badger is designed to use available system memory for filesystem caching. While this behavior is expected and does not negatively impact system performance, it can make collectors appear to consume more memory than anticipated and harder to monitor. To reduce potential confusion around reported memory usage, consider switching to Pebble or Bolt.

Bolt is currently the default persistent queue storage backend, but this may change in the future.

Bindplane supports three persistent queue backends that write to local disk when destinations cannot keep up, each comes with its own benefits and drawbacks.

Which Storage Backend Should You Use?

Deployment Type	Recommended	Why
Single collector	Pebble	Excellent performance with limited resource impact
2+ collector gateway	Pebble	Good performance with anticipated resource usage
Edge/collector network	Pebble	Low resource usage across many distributed deployments
Resource-constrained	Pebble	Minimal memory and CPU overhead
Existing File Storage (Bolt) Queue	Stay or upgrade to Pebble	File Storage still works; upgrade if experiencing issues or would like a performance boost.

Pebble (Recommended)

Pebble is the recommended storage backend for most Bindplane deployments. It provides excellent performance with minimal configuration.

Use Pebble for:

• Balanced disk I/O
• Medium to high throughput workloads
• Smaller disk footprints for your telemetry

See Pebble for configuration options.

Bolt

Bolt is the original persistent queue backend and remains fully supported for the short term. However, Pebble offers in most cases better performance for persistent queueing.

Use Bolt for:

• Existing deployments already using it successfully

See Bolt for more information and upgrade guidance.

Quick Decision Guide

Are you having performance issues?

• Lock contention errors in logs: Upgrade from Bolt to Pebble
• Queue not draining: Upgrade from Bolt to Pebble

Next Steps

1. Review your current deployment
2. Select the appropriate persistent queue for your desired outcome
3. Ensure to follow the Configuration guide in order to not lose data during the transition
4. Monitor performance and resource usage

For detailed configuration information:

• Pebble
• Bolt