AWS S3 Rehydration

The AWS S3 Rehydration source reads previously exported OTLP data back out of an AWS S3 bucket and replays it through a Bindplane pipeline. It processes objects whose paths fall within a configured UTC time range, so you can backfill or reprocess telemetry that was archived to S3 (for example, data written by the AWS S3 destination).

Rehydrated objects must be in OTLP JSON format. The receiver detects and handles both gzip-compressed and uncompressed objects automatically.

Supported Telemetry Types

Platform	Metrics	Logs	Traces
Linux	✓	✓	✓
Windows	✓	✓	✓
macOS	✓	✓	✓
Kubernetes Cluster	✓	✓	✓

Prerequisites

• An AWS S3 bucket containing OTLP JSON objects (optionally gzip-compressed).
• AWS credentials with read access to the bucket. The receiver also needs delete access if Delete on Read is enabled.
• The collector able to reach the AWS S3 API for the configured region.
• When deploying on Kubernetes, select StatefulSet as the deployment method so checkpoint state persists across restarts.

Configuration

General

Parameter	Type	Default	Description
Choose Telemetry Type	Telemetry Selector	Logs, Metrics, Traces	Select which types of telemetry to rehydrate. A separate receiver is configured for each selected type.
Region	Enum	us-east-1	The AWS recognized region string for the bucket. Required.
Bucket	String		Name of the S3 bucket to rehydrate telemetry from. Required.
Folder Prefix	String	""	Root directory (key prefix) of the bucket to rehydrate telemetry from.
Starting Time	Timestamp		The UTC start time for rehydration, in the format YYYY-MM-DDTHH:MM. Required.
Ending Time	Timestamp		The UTC end time for rehydration, in the format YYYY-MM-DDTHH:MM. Required.

Advanced

Parameter	Type	Default	Description
Delete on Read	Boolean	false	If true, objects are deleted after being rehydrated.
Enable Checkpoint	Boolean	true	Enable to use a storage extension that checkpoints rehydration progress, allowing the source to resume after a restart.
Poll Size	Integer	1000	The max number of object descriptions returned by a single poll against the S3 API.
Batch Size	Integer	100	The max number of object descriptions to process and rehydrate at once after retrieving them from the S3 API.
Checkpoint Directory	String	${OIQ_OTEL_COLLECTOR_HOME}/storage	Directory for storing rehydration checkpoint state, useful for resuming operations after disruptions. Relevant only when checkpointing is enabled (storage_enable is true).

Example Configuration

This configuration rehydrates logs, metrics, and traces from a bucket over a fixed UTC time range.

Adjust the Region, Bucket, Starting Time, and Ending Time to match your configuration.

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: aws_s3_rehydration_v2
  name: aws_s3_rehydration_v2
spec:
  type: aws_s3_rehydration_v2
  parameters:
    - name: telemetry_types
      value: ['Logs', 'Metrics', 'Traces']
    - name: region
      value: 'us-east-1'
    - name: bucket
      value: 'my-telemetry-archive'
    - name: prefix
      value: ''
    - name: starting_time
      value: '2026-06-01T00:00'
    - name: ending_time
      value: '2026-06-02T00:00'
    - name: storage_enable
      value: 'true'
    - name: storage_directory
      value: '${OIQ_OTEL_COLLECTOR_HOME}/storage'

Configuration Tips

Scoping the rehydration

• Set starting_time and ending_time to the smallest window that covers the data you need. The receiver only processes objects whose paths fall within this UTC range, so a tight window reduces the number of S3 objects scanned.
• Use prefix to limit rehydration to a single folder within the bucket when the bucket holds more than one data set.

Checkpointing and resuming

• Leave storage_enable on so progress is checkpointed to storage_directory. If the collector restarts mid-rehydration, it resumes from the last checkpoint instead of starting over.
• Enable delete_on_read only when you intend to remove rehydrated objects from the bucket. The AWS credentials must include delete permission for this to succeed.

AWS authentication

• On Linux, Windows, and macOS, the receiver uses the standard AWS credential chain. Provide credentials through the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, a shared credentials file (~/.aws/credentials), or an attached IAM role when running on EC2.
• On Kubernetes, set credentials explicitly in the deployment manifest. Edit the manifest downloaded from Bindplane's agent install page and add the following environment variables to the opentelemetry-container container:

spec:
  template:
    spec:
      containers:
        - name: opentelemetry-container
          env:
            - name: AWS_ACCESS_KEY_ID
              value: '<your access key>'
            - name: AWS_SECRET_ACCESS_KEY
              value: '<your secret key>'

Troubleshooting

Symptom: No data is rehydrated, even though objects exist in the bucket. Solution: Confirm starting_time and ending_time are in UTC and use the YYYY-MM-DDTHH:MM format, and that they bracket the object paths you expect. Also verify the region matches the bucket's region and that prefix points at the correct folder.

Symptom: The receiver logs access-denied or permission errors against the S3 API. Solution: Check that the AWS credentials reachable by the collector have read access to the bucket. If delete_on_read is enabled, the credentials also need delete permission. On Kubernetes, confirm the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set on the opentelemetry-container.

Symptom: Rehydration restarts from the beginning after a collector restart. Solution: Ensure storage_enable is true and that storage_directory is writable and persists across restarts. On Kubernetes, deploy with StatefulSet so the checkpoint volume survives pod restarts.

Related Resources

• Amazon S3 documentation
• Amazon S3 buckets
• AWS S3 destination