Persistent Queue

Persistent Queue storage buffers telemetry data temporarily, preventing data loss when destinations are unavailable or during collector maintenance.

Bolt is currently the default persistent queue storage backend, but this may change in the future.

Bindplane supports three persistent queue backends that write to local disk when destinations cannot keep up, each comes with its own benefits and drawbacks.

Which Storage Backend Should You Use?

Deployment Type	Recommended	Why
Single collector	Pebble	Excellent performance with limited resource impact
2-3 collector gateway	Pebble	Good performance with anticipated resource usage
5+ collector gateway	Badger	Higher throughput per collector makes large-scale deployments with a larger amount of throughput more efficient
Edge/agent network	Pebble	Low resource usage across many distributed deployments
Resource-constrained	Pebble	Minimal memory and CPU overhead
Existing File Storage (Bolt) Queue	Stay or upgrade to Pebble	File Storage still works; upgrade if experiencing issues or would like a performance boost.

Pebble (Recommended)

Pebble is the recommended storage backend for most Bindplane deployments. It provides excellent performance with minimal configuration.

Use Pebble for:

• Balanced disk I/O

• Medium to high throughput workloads

• Smaller disk footprints for your telemetry

See Pebble for configuration options.

Badger

Badger is optimized for large-scale deployments. It trades increased memory usage for better per-collector throughput.

Use Badger for:

• Deployments where per-collector throughput is important

• When infrastructure cost savings from fewer collectors justify higher memory use

See Badger for configuration options.

Bolt

Bolt is the original persistent queue backend and remains fully supported for the short term. However, Pebble and Badger offer in most cases better performance for persistent queueing.

Use Bolt for:

• Existing deployments already using it successfully

See Bolt for more information and upgrade guidance.

Quick Decision Guide

Ask yourself:

1. Are you having performance issues?

   • Lock contention errors in logs: Upgrade from Bolt to Pebble

   • Queue not draining: Upgrade from Bolt to Pebble

   • Throughput bottleneck: Switch from Pebble to Badger

2. What's your memory allocation?

   • Generally at Bindplane recommended sizing (8GB): Start with Pebble

   • Above minimum and need high throughput (12GB+): Start with Badger

   • Severely constrained resources (< 8GB): Use Pebble

Next Steps

1. Review your current deployment

2. Select the appropriate persistent queue for your desired outcome

3. Ensure to follow the Configuration guide in order to not lose data during the transition

4. Monitor performance and resource usage

For detailed configuration information:

• Pebble

• Badger

• Bolt