File Storage

File Storage is a collector-level capability that persists data to a file on the local filesystem. It does not receive, process, or export telemetry. Instead, other components use it as a backing store for state that must survive a collector restart. Under the hood it uses bbolt, an embedded key/value store that writes to a single on-disk file, and each storage client gets its own database in the configured directory.

This is what makes a persistent queue durable: when the Persistent Queue or a destination's sending queue is backed by File Storage, telemetry that is buffered when the collector stops is still on disk when it starts again, so nothing in the queue is lost. Stateful processors such as the Drain processor likewise use File Storage to retain learned state (for Drain, the learned log templates) across restarts.

Configuration

Basic Configuration

Storage

Parameter	Type	Required	Default	Description
Directory	String	Yes	${OIQ_OTEL_COLLECTOR_HOME}/storage	The directory where the storage database files are written.
Create Directory	Boolean	No	true	Whether to create the storage directory if it does not already exist.
Synchronize to Disk	Boolean	No	false	Whether to synchronize writes to disk after each operation. This helps ensure database integrity if the process is interrupted, at the cost of performance.

Advanced

Parameter	Type	Required	Default	Description
Timeout	Duration	No	1	The maximum time to wait for a file lock, in seconds.
Enable Compaction	Boolean	No	false	Compact the storage database to reclaim disk space. The remaining compaction parameters apply only when this is enabled.
Compaction Directory	String	Yes	${OIQ_OTEL_COLLECTOR_HOME}/storage	The directory used to store the temporary file while compacting. Relevant when Enable Compaction is true.
Compact on Start	Boolean	No	false	Whether to compact the database when the collector starts. Relevant when Enable Compaction is true.
Compact on Rebound	Boolean	No	true	Whether to perform online compaction when the database size rebounds below the configured thresholds. Relevant when Enable Compaction is true.
Max Transaction Size	Integer	No	65536	The maximum number of items to move during a single compaction transaction. Relevant when Enable Compaction is true.
Rebound Needed Threshold (MiB)	Integer	No	100	The database size, in MiB, that must be exceeded to flag the database for rebound compaction. Relevant when Enable Compaction and Compact on Rebound are true.
Rebound Trigger Threshold (MiB)	Integer	No	10	The database size, in MiB, that the storage must shrink below to trigger rebound compaction. Relevant when Enable Compaction and Compact on Rebound are true.
Check Interval	Duration	No	5	How often, in seconds, to check whether rebound compaction conditions are met. Relevant when Enable Compaction and Compact on Rebound are true.

Examples

Durable storage with rebound compaction

This example points File Storage at a dedicated directory and turns on compaction so the database periodically reclaims unused space. It exercises the advanced parameters Timeout, Enable Compaction, Compaction Directory, Compact on Rebound, Rebound Needed Threshold, Rebound Trigger Threshold, and Check Interval.

apiVersion: bindplane.observiq.com/v1
kind: Extension
metadata:
  name: file-storage
spec:
  type: file_storage
  parameters:
    - name: directory
      value: /var/lib/otel/storage
    - name: create_directory
      value: true
    - name: fsync
      value: true
    - name: timeout
      value: 5
    - name: enable_compaction
      value: true
    - name: compaction_directory
      value: /var/lib/otel/compaction
    - name: compaction_on_rebound
      value: true
    - name: compaction_rebound_needed_threshold_mib
      value: 200
    - name: compaction_rebound_trigger_threshold_mib
      value: 20
    - name: compaction_check_interval
      value: 10

Configuration Tips

• Use a directory on persistent, fast local storage, not a tmpfs or network mount. The whole point of File Storage is that the file outlives the collector process, and bbolt is sensitive to network filesystem locking semantics.
• Enable Synchronize to Disk when data integrity after an abrupt crash matters more than throughput. It calls fsync after each write, which is safer but slower.
• Turn on compaction for long-running collectors whose queue depth varies widely. Compaction reclaims space the database grew into during traffic spikes instead of leaving it allocated on disk.

Troubleshooting

The collector fails to start with a directory or permission error

Symptoms: the collector logs an error opening the storage database, or reports the storage directory cannot be created.

Solutions:

1. Confirm the configured Directory exists, or set Create Directory to true so the collector creates it.
2. Verify the collector's user has read/write permission on the directory and that the path is not on a read-only or network mount.

Disk usage keeps growing

Symptoms: the storage file grows over time and does not shrink after queue depth falls.

Solutions:

1. Enable Compaction so the database reclaims space that is no longer referenced.
2. With Compact on Rebound enabled, tune Rebound Needed Threshold and Rebound Trigger Threshold so compaction actually triggers for your workload's database size.

Standalone Extension

apiVersion: bindplane.observiq.com/v1
kind: Extension
metadata:
  name: file-storage
spec:
  type: file_storage
  parameters:
    - name: directory
      value: /var/lib/otel/storage

Related Resources

• Persistent Queue — back a persistent queue with File Storage so buffered telemetry survives collector restarts.
• Drain — a stateful processor that uses File Storage to retain learned log templates across restarts.