> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/processors/cumulative-to-delta.md). # Cumulative To Delta ## Cumulative to Delta Converts cumulative sum, histogram, and exponential histogram metrics to delta aggregation temporality. The processor remembers the last value it saw for each metric series and emits the difference on each new data point. Only monotonic sums are converted. Non-monotonic sums, gauges, and metrics that are already delta pass through unchanged. Operates on metrics only. #### Supported Telemetry Types | Metrics | Logs | Traces | | ------- | ---- | ------ | | ✓ | | | #### Where to Run This Processor This processor is stateful. It computes each delta by subtracting the previous value it saw for a series, so it is only accurate when every data point for that series reaches the same collector instance. It produces its most useful output when the collector lifecycle is tied to the metric source. Place it as close to the edge as possible. | Placement | Recommendation | Why | | --------------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Agent on the host or node emitting the metric | Recommended | One collector sees every point for each series. State is complete and restarts of the source and collector are visible to the processor. | | Single gateway with a fixed set of sources | Acceptable | Works as long as each source always sends to this one gateway. Use `auto` or `drop` for Initial Value so a gateway restart does not emit one large delta per series. | | Multiple gateways behind a load balancer | Avoid | A source's points are spread across gateways. Each gateway holds its own partial state and emits deltas computed from the wrong previous value. The result is silently inaccurate data, not an error. | If you must convert in a gateway tier, route each source to exactly one gateway and accept that a gateway restart resets state for every series it tracks. In most environments the simpler fix is to move the processor to the collector configuration and let gateways forward the already converted delta metrics. #### Configuration **Include** | Parameter | Type | Required | Default | Description | | ------------ | ------------------------------------------- | -------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric Names | Strings | No | *(empty)* | Metric names to convert. If empty, all metrics are converted unless excluded. With `regexp`, each entry is a [re2](https://github.com/google/re2/wiki/Syntax) pattern. | | Match Type | Enum: strict, regexp | No | strict | How the metric names are matched. `strict` requires an exact metric-name match. `regexp` matches each entry as a regular expression. | | Metric Types | Enums: sum, histogram, exponentialhistogram | No | *(empty)* | Metric types to convert. If empty, all supported types are converted. | **Exclude** | Parameter | Type | Required | Default | Description | | ------------ | ------------------------------------------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------ | | Metric Names | Strings | No | *(empty)* | Metric names to leave as cumulative. With `regexp`, each entry is a [re2](https://github.com/google/re2/wiki/Syntax) pattern. | | Match Type | Enum: strict, regexp | No | strict | How the metric names are matched. `strict` requires an exact metric-name match. `regexp` matches each entry as a regular expression. | | Metric Types | Enums: sum, histogram, exponentialhistogram | No | *(empty)* | Metric types to leave as cumulative. | **Advanced** | Parameter | Type | Required | Default | Description | | ------------- | ---------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Max Staleness | Duration | Yes | 1h | Time after which a metric's previous state is forgotten if no new data point arrives. `0` keeps state indefinitely. Example: `1h`. | | Initial Value | Enum: auto, keep, drop | No | auto | How to handle the first data point of each series. `auto` drops it unless the start time indicates it is the true first point. `keep` always emits the first observed value as a delta. `drop` always discards it. | If a metric matches both Include and Exclude, Exclude takes precedence. If neither Include nor Exclude is set, every monotonic sum, histogram, and exponential histogram is converted. #### Examples **Convert all cumulative metrics** With no Include or Exclude settings, every monotonic sum, histogram, and exponential histogram flowing through the pipeline is converted to delta. ```yaml apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: name: cumulative-to-delta spec: type: cumulative_to_delta parameters: [] ``` **Convert only sums matching a name pattern** Convert every cumulative sum whose name starts with `system.cpu.` to delta. Histograms and metrics outside that prefix remain cumulative. ```yaml apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: name: cumulative-to-delta spec: type: cumulative_to_delta parameters: - name: include_metrics value: - "^system\\.cpu\\..*$" - name: include_match_type value: regexp - name: include_metric_types value: - sum ``` **Convert everything except specific metrics** Convert all supported metrics, but leave two named counters and all exponential histograms cumulative. Because the collector is co-located with the metric source, the first observed value is kept. ```yaml apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: name: cumulative-to-delta spec: type: cumulative_to_delta parameters: - name: exclude_metrics value: - system.network.io - system.disk.operations - name: exclude_match_type value: strict - name: exclude_metric_types value: - exponentialhistogram - name: max_staleness value: 30m - name: initial_value value: keep ``` #### Configuration Tips * When you add this processor and inspect it in the processor node view, the converted metrics appear to be deleted. This is expected. The cumulative series is replaced by a delta series, and the new metric is visible on the next downstream processor node. * Choose Initial Value based on where the processor runs. `auto` is the safe default. It drops the first point of a series unless the start time shows the counter was just created, so a collector restart does not emit one large delta covering everything the counter accumulated before. Use `keep` on an agent or sidecar whose lifecycle is tied to the metric source, where the first observation really is the first value. Use `drop` in a gateway when you must guarantee no delta includes values observed before the collector started. * Set Max Staleness comfortably above the reporting interval of your slowest metric. If a series goes quiet longer than Max Staleness, its state is forgotten and the next point is treated as a first point again. Setting it to `0` never expires state, which grows memory for every series ever seen. * With `strict` match type each entry must be the full metric name. With `regexp` each entry is a re2 pattern. Anchor patterns (for example `^system\.cpu\.`) to avoid unintended partial matches, and escape literal dots. * Delta temporality is required or preferred by some destinations and rejected by others. Confirm what your destination expects before converting. #### Troubleshooting **The first data point of each metric is missing** Symptoms: a converted metric has no value for its first reporting interval after the collector starts, or after it has been quiet for a while. Solutions: 1. With Initial Value set to `auto` or `drop`, the first point of each series is discarded by design because there is no previous value to subtract from. Switch to `keep` if the collector is co-located with the metric source and the first observation is the true starting value. 2. If a metric disappears and reappears, check Max Staleness. A series that is idle for longer than Max Staleness loses its state and its next point is treated as a first point. **Deltas are wrong, negative, or spike unexpectedly** Symptoms: converted values do not match the change in the cumulative counter, or a single very large delta appears. Solutions: 1. Confirm every data point for a series reaches the same collector. This is almost always a multi-gateway deployment. Move the processor to the agent, or pin each source to one gateway. See Where to Run This Processor. 2. A large spike immediately after a collector restart usually means Initial Value is `keep`. Switch to `auto` so the first observation after a restart is dropped. 3. A cumulative counter that resets on the source side (for example the process restarted) causes the processor to drop the data point where the decrease is detected rather than emit a negative delta. Deltas resume on the following point. A recurring gap of one point is expected whenever the source restarts. **A metric was not converted** Symptoms: a metric still arrives at the destination with cumulative temporality. Solutions: 1. Only monotonic sums, histograms, and exponential histograms are converted. Gauges and non-monotonic sums are never converted and pass through unchanged. 2. If the metric matches an Exclude rule it is left cumulative even when it also matches Include. Exclude takes precedence. 3. If Include Metric Names is set, only listed metrics are converted. With `strict` match type the name must match exactly, including casing and the full dotted name. Switch to `regexp` if you need pattern matching. 4. If Include Metric Types is set, only the listed types are converted. Add the missing type or clear the list to convert all supported types. **The processor appears to delete metrics in the preview** Symptoms: the processor node view shows the converted metrics as removed. Solutions: 1. This is how the conversion is displayed. The cumulative series is removed and a delta series is added. Select the next downstream processor node to see the converted metric. #### Standalone Processor ```yaml apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: name: cumulative-to-delta spec: type: cumulative_to_delta parameters: - name: include_metric_types value: - sum - name: max_staleness value: 1h - name: initial_value value: auto ``` #### Related Resources * [Cumulative to Delta Processor — OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/cumulativetodeltaprocessor/README.md), the upstream processor this resource configures. * [Statefulness warning — OpenTelemetry Collector standard warnings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/standard-warnings.md#statefulness) * [re2 regular-expression syntax](https://github.com/google/re2/wiki/Syntax) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.bindplane.com/integrations/processors/cumulative-to-delta.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.