> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/destinations/aws-s3.md). # AWS S3 The AWS S3 destination exports telemetry from a Bindplane pipeline as objects written to an Amazon S3 bucket. It supports logs, metrics, and traces, writing each signal as timestamped, partitioned files under a configurable prefix using the OpenTelemetry [`awss3exporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awss3exporter/README.md). ### Supported Telemetry | Metrics | Logs | Traces | | ------- | ---- | ------ | | ✓ | ✓ | ✓ | ### Prerequisites * A pre-created S3 bucket in the AWS account and region you intend to export to. The exporter does not create the bucket. * The AWS region the bucket resides in. * AWS credentials available to the collector with permission to write objects to that bucket. At minimum the credentials need the `s3:PutObject` action on the bucket's objects (`arn:aws:s3:::/*`). See the AWS [identity-based policy examples for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html) and the [`PutObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) API reference. A minimal IAM policy granting object writes to a single bucket: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "BindplaneS3Export", "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::my-telemetry-bucket/*" } ] } ``` If objects are encrypted with SSE-KMS, the credentials also need `kms:GenerateDataKey` and `kms:Decrypt` on the key. #### Providing credentials to the collector The exporter resolves credentials using the standard AWS SDK credential chain. Configure them on the host running the collector with one of: * The `aws` CLI ([AWS CLI getting started](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)). The collector runs as root by default, so install and configure the CLI under the collector's system account. * AWS [environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION`), for example via a `systemd` override. * An instance role / IRSA when running on EC2, ECS, or EKS. ### Configuration
Bindplane docs - AWS S3 - image 1
**Bucket and routing** | Parameter | Type | Required | Default | Description | | --------------- | ------------------ | -------- | --------------------- | -------------------------------------------------------------------------------------------------- | | Telemetry Types | Telemetry Selector | No | Logs, Metrics, Traces | Which signals this destination exports. | | Region | Enum (creatable) | Yes | us-east-1 | The AWS region of the bucket. Choose from the listed regions or enter another valid region string. | | Bucket | String | Yes | *(empty)* | Name of the S3 bucket to export telemetry into. | | Folder Prefix | String | No | *(empty)* | Root directory within the bucket to export telemetry into. | | File Prefix | String | No | *(empty)* | Prefix for the name of exported telemetry files. | **Object format and storage (Advanced)** | Parameter | Type | Required | Default | Description | | ------------------------ | ------------------ | -------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Granularity | Enum: minute, hour | No | minute | Granularity of the timestamps in the S3 key. Used with AWS S3 exporter versions before v0.121.0; v0.121.0 and later use Partition Format instead. | | Partition Format | String | No | `year=%Y/month=%m/day=%d/hour=%H/minute=%M` | `strftime` format defining the S3 key partition. Available with the AWS S3 exporter v0.121.0 and newer. See the [strftime reference](https://www.man7.org/linux/man-pages/man3/strftime.3.html). | | Compression | Enum: none, gzip | No | none | Whether data is sent and stored in S3 using gzip compression. | | Storage Class | Enum | No | STANDARD | S3 storage class for written objects.1 | | Endpoint | String | No | *(empty)* | Overrides the endpoint used to connect to S3 instead of constructing it from the region and bucket. Generally not needed. | | Resource Label to Prefix | String | No | `com.awss3.prefix` | An S3 prefix dynamically derived from the named resource attribute's value. | 1. *One of `STANDARD`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `EXPRESS_ONEZONE`, `GLACIER_IR`, `GLACIER`, `DEEP_ARCHIVE`. Available in agents v1.71.5 and later. See* [*S3 storage classes*](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html#sc-compare)*.* **Marshalers (Advanced)** The marshaler determines the on-disk format of exported objects. Defaults to `otlp_json` for every signal. | Parameter | Type | Required | Default | Description | | ----------------- | --------------------------------------------- | -------- | ---------- | ----------------------------------------------------------------------------------------------------------- | | Drop Raw Copy | Boolean | No | true | When enabled, drops the raw copy of the log stored in `log.record.original` before export. Applies to logs. | | Logs Marshaler | Enum: otlp\_json, otlp\_proto, sumo\_ic, body | No | otlp\_json | Format of log data sent to S3. `sumo_ic` and `body` are logs-only. Shown when Logs is selected. | | Metrics Marshaler | Enum: otlp\_json, otlp\_proto | No | otlp\_json | Format of metric data sent to S3. Shown when Metrics is selected. | | Traces Marshaler | Enum: otlp\_json, otlp\_proto | No | otlp\_json | Format of trace data sent to S3. Shown when Traces is selected. | **Timeout and Retry (Advanced)** | Parameter | Type | Required | Default | Description | | --------------- | ----------------------------- | -------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | Timeout | Integer (seconds) | No | 30 | Maximum time to wait for a batch to be sent to the destination. | | Retry Mode | Enum: standard, adaptive, nop | No | standard | The retryer implementation. Set to `nop` to disable retries. | | Maximum Retries | Integer | No | 3 | Maximum number of times to retry a request. 0 retries all retryable errors until success or a non-retryable error. Shown when Retry Mode is not `nop`. | | Maximum Backoff | Integer (seconds) | No | 20 | Maximum time to wait between retries. Shown when Retry Mode is `standard`. | **Sending Queue (Advanced)** | Parameter | Type | Required | Default | Description | | ------------------------- | --------- | -------- | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Enable Sending Queue | Boolean | No | false | Buffer telemetry temporarily before sending to reduce loss during a temporary network outage. See the [exporterhelper reference](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md). | | Number of Consumers | Integer | No | 10 | Number of consumers that dequeue batches. Shown when the sending queue is enabled. | | Queue Size | Integer | No | 5000 | Maximum number of batches the queue can hold. Shown when the sending queue is enabled. | | Wait for Result | Boolean | No | false | Block incoming requests until the request is processed. Not compatible with persistent queuing. Shown when the sending queue is enabled. | | Block on Overflow | Boolean | No | false | If true, block the request until the queue has space; otherwise reject the data immediately. Shown when the sending queue is enabled. | | Enable Persistent Queuing | Boolean | No | true | Buffer telemetry to disk so it survives network outages and collector restarts. Shown when the sending queue is enabled and Wait for Result is false. | | Persistent Queue Storage | Extension | Yes | File storage (`${OIQ_OTEL_COLLECTOR_HOME}/storage`) | Storage extension backing the persistent queue. See [persistent queue](https://docs.bindplane.com/configuration/bindplane-otel-collector/persistent-queue). Shown when the sending queue and persistent queuing are enabled and Wait for Result is false. | ### Examples #### Export all signals to a partitioned bucket This destination writes logs, metrics, and traces to the `my-telemetry-bucket` bucket in `us-east-1`, under a `bindplane/` prefix, using gzip compression. ```yaml apiVersion: bindplane.observiq.com/v1 kind: Destination metadata: name: aws-s3 spec: type: aws_s3 parameters: - name: telemetry_types value: - Logs - Metrics - Traces - name: region value: us-east-1 - name: bucket value: my-telemetry-bucket - name: prefix value: bindplane - name: compression value: gzip ``` ### Configuration Tips * Set Region to the region the bucket actually lives in. A region mismatch produces authorization or endpoint errors at export time. * Use gzip Compression and a colder Storage Class (for example STANDARD\_IA or GLACIER\_IR) to lower storage cost when objects are written for archival rather than frequent reads. * For lossless delivery across collector restarts, keep the sending queue and persistent queuing enabled (the default). Wait for Result is incompatible with persistent queuing, so leave it off when persistence matters. ### Troubleshooting #### Access denied writing objects Symptoms: the collector logs `AccessDenied` or 403 errors from S3 and no objects appear. Solutions: 1. Confirm the credentials the collector resolves have `s3:PutObject` on `arn:aws:s3:::/*`. 2. If the bucket enforces SSE-KMS, grant `kms:GenerateDataKey` and `kms:Decrypt` on the key. 3. Verify the credentials are installed under the account the collector runs as (root by default). #### No objects in the bucket / wrong region Symptoms: exports appear to succeed but the bucket is empty, or requests fail with endpoint errors. Solutions: 1. Confirm Region matches the bucket's region. 2. Confirm the Bucket name is exact and exists in that account. 3. If an Endpoint override is set, confirm it is correct, otherwise clear it to let the region and bucket construct the endpoint. #### Data dropped during outages Symptoms: telemetry gaps coincide with network interruptions or collector restarts. Solutions: 1. Enable the sending queue and persistent queuing so batches buffer to disk and replay on recovery. 2. Increase Queue Size if bursts overflow the queue. ### Standalone Destination ```yaml apiVersion: bindplane.observiq.com/v1 kind: Destination metadata: name: aws-s3 spec: type: aws_s3 parameters: - name: telemetry_types value: - Logs - Metrics - Traces - name: region value: us-east-1 - name: bucket value: my-telemetry-bucket ``` ### Related Resources * [`awss3exporter` — OpenTelemetry Collector Contrib reference](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awss3exporter/README.md) * [Identity-based policy examples for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.bindplane.com/integrations/destinations/aws-s3.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.