# AWS Security Lake

{% hint style="info" %}
This destination is in **Alpha** stability.
{% endhint %}

## Supported Types

| Logs | Metrics | Traces |
| ---- | ------- | ------ |
| ✓    |         |        |

The AWS Security Lake destination exports OCSF-formatted logs as Parquet files to an Amazon Security Lake S3 bucket, partitioned by account, region, and OCSF class. This destination requires the [OCSF Standardization processor](https://docs.bindplane.com/integrations/processors) upstream in the pipeline.

## Prerequisites

* Logs must be transformed into OCSF format before reaching this destination. Add the OCSF Standardization processor upstream in your pipeline.
* An [AWS Security Lake custom source](https://docs.aws.amazon.com/security-lake/latest/userguide/adding-custom-sources.html) must be registered in your AWS account for each OCSF class you intend to export.
* AWS credentials must be available to the collector via the standard AWS credential chain (environment variables, shared credentials file, or IAM role).

## Configuration

| Parameter         | Type     | Description                                                                                                                                                                                         |
| ----------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| region\*          | `enum`   | The AWS region where the Security Lake S3 bucket resides.                                                                                                                                           |
| s3\_bucket\*      | `string` | The name of the Security Lake S3 bucket.                                                                                                                                                            |
| account\_id\*     | `string` | The AWS account ID used in the S3 partition path.                                                                                                                                                   |
| ocsf\_version\*   | `enum`   | The OCSF schema version to use for Parquet output. Supported values: `1.0.0`, `1.1.0`, `1.2.0`, `1.3.0`.                                                                                            |
| custom\_sources\* | `map`    | A mapping of custom source names registered in Security Lake to their OCSF class IDs. The key is the custom source name and the value is the integer OCSF class ID. At least one entry is required. |

<mark style="color:red;">\*</mark>*<mark style="color:red;">required field</mark>*

### **Advanced Configuration**

| Parameter      | Type     | Default | Description                                                                                          |
| -------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------- |
| role\_arn      | `string` |         | An optional IAM role ARN to assume for S3 writes.                                                    |
| endpoint       | `string` |         | An optional custom endpoint for S3 writes. Overrides the default AWS endpoint. Generally not needed. |
| timeout        | `int`    | `5`     | Timeout in seconds for S3 write operations.                                                          |
| batch\_size    | `int`    | `10000` | Number of events to buffer before flushing to S3.                                                    |
| batch\_timeout | `int`    | `5`     | Maximum time in minutes to wait before flushing to S3, regardless of batch size.                     |

> *`batch_size` applies across all OCSF classes handled by a single destination instance. To buffer independently per class ID, configure a separate destination instance for each class, each with a single entry in `custom_sources`.*

### **Retry and Queuing**

This destination supports the retry settings, the sending queue settings, and the persistent queue settings.

| Sending Queue | Persistent Queue | Retry on Failure |
| ------------- | ---------------- | ---------------- |
| ✓             | ✓                | ✓                |

## How It Works

The destination processes each batch of OCSF-formatted logs through the following steps:

1. **Route by class** - The `class_uid` field is read from each log record's body and looked up against the configured custom sources. Records with a `class_uid` that doesn't match any configured source are skipped with a warning (see Dropped Records).
2. **Partition** - Matching records are grouped into partitions by custom source name, OCSF class ID, and event day (derived from the `time` field in the log body). Each partition becomes a separate Parquet file.
3. **Serialize to Parquet** - Records within each partition are sorted by `time` ascending, then serialized to Parquet using the OCSF schema for the configured `ocsf_version`, with ZSTD compression.
4. **Upload to S3** - Each Parquet file is uploaded to the Security Lake bucket using the following key format:

```
ext/{sourceName}/region={region}/accountId={accountID}/eventDay={YYYYMMDD}/{sourceName}_{timestamp}_{uuid}.parquet
```

## Dropped Records

Log records that are not in valid OCSF format are silently dropped. A warning is logged for each dropped record. Ensure the OCSF Standardization processor is upstream in the pipeline to avoid data loss.

## Credentials

With AWS Security Lake, you must provide AWS credentials that allow `s3:PutObject` access to the Security Lake S3 bucket. There are two ways to configure this:

1. Using the AWS CLI to set up a credentials profile.
2. Specifying environment variables with access keys.

The [AWS CLI getting started guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) will instruct you how to install it for your current user or all users.

The Bindplane OTel Collector runs [as root](https://github.com/observIQ/bindplane-otel-collector/blob/main/docs/installation-linux.md#configuring-the-collector) by default, meaning the AWS CLI and credentials should be installed under the collector system's root account.

### **Environment Variables**

Alternatively, [AWS environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) can be specified to override a credentials file. You can modify the collector's environment variables by configuring a `systemd` override. Run `sudo systemctl edit observiq-otel-collector` and add your access key, secret key, and region:

```
[Service]
Environment=AWS_ACCESS_KEY_ID=******
Environment=AWS_SECRET_ACCESS_KEY=******
Environment=AWS_DEFAULT_REGION=us-east-1
```

After making that change, reload Systemd and restart the collector service.

```bash
sudo systemctl daemon-reload
sudo systemctl restart observiq-otel-collector
```

## Example Configuration

### **Web Interface**

<figure><img src="https://1405008107-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgmiOMzBfoNFwmKJFHMcJ%2Fuploads%2FAyUDrX7J6WREvLoNH7KR%2Fbpss_Screenshot%202026-04-01%20at%2010.04.27%E2%80%AFAM.png?alt=media&#x26;token=0c6fe690-b395-4c7d-82c6-927ce6f1d6fe" alt=""><figcaption></figcaption></figure>

### **Standalone Destination**

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  name: aws-security-lake
spec:
  type: aws_security_lake
  parameters:
    - name: region
      value: us-east-1
    - name: s3_bucket
      value: aws-security-data-lake-us-east-1-xxxxxxxxxxxx
    - name: account_id
      value: '123456789012'
    - name: ocsf_version
      value: '1.3.0'
    - name: custom_sources
      value:
        AuthenticationEvents: '3002'
        FSActivity: '1001'
```