> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/extensions/file-storage.md). # File Storage File Storage is a collector-level capability that persists data to a file on the local filesystem. It does not receive, process, or export telemetry. Instead, other components use it as a backing store for state that must survive a collector restart. Under the hood it uses [bbolt](https://github.com/etcd-io/bbolt), an embedded key/value store that writes to a single on-disk file, and each storage client gets its own database in the configured directory. This is what makes a persistent queue durable: when the [Persistent Queue](/configuration/bindplane-otel-collector/persistent-queue.md) or a destination's sending queue is backed by File Storage, telemetry that is buffered when the collector stops is still on disk when it starts again, so nothing in the queue is lost. Stateful processors such as the [Drain processor](/integrations/processors/drain.md) likewise use File Storage to retain learned state (for Drain, the learned log templates) across restarts. ### Configuration #### Basic Configuration **Storage** | Parameter | Type | Required | Default | Description | | ------------------- | ------- | -------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Directory | String | Yes | `${OIQ_OTEL_COLLECTOR_HOME}/storage` | The directory where the storage database files are written. | | Create Directory | Boolean | No | `true` | Whether to create the storage directory if it does not already exist. | | Synchronize to Disk | Boolean | No | `false` | Whether to synchronize writes to disk after each operation. This helps ensure database integrity if the process is interrupted, at the cost of performance. | **Advanced** | Parameter | Type | Required | Default | Description | | ------------------------------- | -------- | -------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Timeout | Duration | No | `1` | The maximum time to wait for a file lock, in seconds. | | Enable Compaction | Boolean | No | `false` | Compact the storage database to reclaim disk space. The remaining compaction parameters apply only when this is enabled. | | Compaction Directory | String | Yes | `${OIQ_OTEL_COLLECTOR_HOME}/storage` | The directory used to store the temporary file while compacting. Relevant when Enable Compaction is `true`. | | Compact on Start | Boolean | No | `false` | Whether to compact the database when the collector starts. Relevant when Enable Compaction is `true`. | | Compact on Rebound | Boolean | No | `true` | Whether to perform online compaction when the database size rebounds below the configured thresholds. Relevant when Enable Compaction is `true`. | | Max Transaction Size | Integer | No | `65536` | The maximum number of items to move during a single compaction transaction. Relevant when Enable Compaction is `true`. | | Rebound Needed Threshold (MiB) | Integer | No | `100` | The database size, in MiB, that must be exceeded to flag the database for rebound compaction. Relevant when Enable Compaction and Compact on Rebound are `true`. | | Rebound Trigger Threshold (MiB) | Integer | No | `10` | The database size, in MiB, that the storage must shrink below to trigger rebound compaction. Relevant when Enable Compaction and Compact on Rebound are `true`. | | Check Interval | Duration | No | `5` | How often, in seconds, to check whether rebound compaction conditions are met. Relevant when Enable Compaction and Compact on Rebound are `true`. | ### Examples #### Durable storage with rebound compaction This example points File Storage at a dedicated directory and turns on compaction so the database periodically reclaims unused space. It exercises the advanced parameters Timeout, Enable Compaction, Compaction Directory, Compact on Rebound, Rebound Needed Threshold, Rebound Trigger Threshold, and Check Interval. ```yaml apiVersion: bindplane.observiq.com/v1 kind: Extension metadata: name: file-storage spec: type: file_storage parameters: - name: directory value: /var/lib/otel/storage - name: create_directory value: true - name: fsync value: true - name: timeout value: 5 - name: enable_compaction value: true - name: compaction_directory value: /var/lib/otel/compaction - name: compaction_on_rebound value: true - name: compaction_rebound_needed_threshold_mib value: 200 - name: compaction_rebound_trigger_threshold_mib value: 20 - name: compaction_check_interval value: 10 ``` ### Configuration Tips * Use a directory on persistent, fast local storage, not a tmpfs or network mount. The whole point of File Storage is that the file outlives the collector process, and bbolt is sensitive to network filesystem locking semantics. * Enable Synchronize to Disk when data integrity after an abrupt crash matters more than throughput. It calls fsync after each write, which is safer but slower. * Turn on compaction for long-running collectors whose queue depth varies widely. Compaction reclaims space the database grew into during traffic spikes instead of leaving it allocated on disk. ### Troubleshooting #### The collector fails to start with a directory or permission error Symptoms: the collector logs an error opening the storage database, or reports the storage directory cannot be created. Solutions: 1. Confirm the configured Directory exists, or set Create Directory to `true` so the collector creates it. 2. Verify the collector's user has read/write permission on the directory and that the path is not on a read-only or network mount. #### Disk usage keeps growing Symptoms: the storage file grows over time and does not shrink after queue depth falls. Solutions: 1. Enable Compaction so the database reclaims space that is no longer referenced. 2. With Compact on Rebound enabled, tune Rebound Needed Threshold and Rebound Trigger Threshold so compaction actually triggers for your workload's database size. ### Standalone Extension ```yaml apiVersion: bindplane.observiq.com/v1 kind: Extension metadata: name: file-storage spec: type: file_storage parameters: - name: directory value: /var/lib/otel/storage ``` ### Related Resources * [Persistent Queue](/configuration/bindplane-otel-collector/persistent-queue.md) — back a persistent queue with File Storage so buffered telemetry survives collector restarts. * [Drain](/integrations/processors/drain.md) — a stateful processor that uses File Storage to retain learned log templates across restarts. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.bindplane.com/integrations/extensions/file-storage.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.