AWS S3 Event

The AWS S3 Event Receiver consumes S3 event notifications for object creation events (s3:ObjectCreated:*) and emits the S3 object as the string body of a log record.

Supported Platforms

Platform	Supported
Linux	✓
Windows	✓
macOS	✓

Available in the Bindplane Distro for OpenTelemetry Collector v1.74.0+.

Prerequisites

• An AWS account with access to S3 and SQS.

• An SQS queue configured to receive S3 event notifications.

1. Ensure the collector has permission to read and delete messages from the SQS queue.

2. Ensure the collector has permission to read objects from the S3 bucket.

How It Works

1. The receiver polls an SQS queue for S3 event notifications.

2. When an object creation event (s3:ObjectCreated:*) is received, the receiver downloads the S3 object.

3. The receiver reads the object into the body of a new log record.

4. Non-object creation events are ignored but removed from the queue.

5. If an S3 object is not found (404 error), the corresponding SQS message is preserved for retry later.

Configuration Fields

Field	Type	Default	Required	Description
sqs_queue_url	string		true	The URL of the SQS queue to poll for S3 event notifications.
standard_poll_interval	duration	15	false	The interval (in seconds) at which the SQS queue is polled for messages.
max_poll_interval	duration	2	false	The maximum interval (in seconds) at which the SQS queue is polled for messages.
polling_backoff_factor	float	2	false	The factor by which the polling interval is multiplied after an unsuccessful poll.
workers	int	5	false	The number of workers to process events.
visibility_timeout	duration	300	false	How long (in seconds) messages received from the queue will be invisible to other consumers.
visibility_extension_interval	duration	1m	false	How often to extend message visibility during processing. Should be less than visibility_timeout. Minimum is 10s.
max_visibility_window	duration	1h	false	Maximum total time a message can remain invisible before becoming visible to other consumers. Must be less than SQS's 12-hour limit.
max_log_size	int	1048576	false	The maximum size of a log record in bytes. Logs exceeding this size will be split
max_logs_emitted	int	1000	false	The maximum number of log records to emit in a single batch. A higher number will result in fewer batches, but more memory.
bucket_name_filter	string		false	When set, the source will only emit logs for bucket names that match the specified regex.
object_key_filter	string		false	When set, the source will only emit logs for object names that match the specified regex.
notification_type	enum	s3	false	Options are s3 and sns. S3 notifications are the default for S3 to SQS. SNS is for S3 notifications sent via SNS to SQS.
enable_offset_storage	bool	false	false	When enabled, the current position into an object will be saved to Redis, and reading will resume from where it left off after a collector restart.
redis_hostname	string		false	The hostname or IP address of the Redis server used for offset storage.
redis_port	int	6379	false	The port number of the Redis server used for offset storage.
redis_password	string		false	The password for the Redis server used for offset storage.
redis_database	string		false	The Redis database number to use for offset storage.
redis_expiration	int	0	false	The expiration time (in seconds) for offset storage in Redis.
redis_tls_enabled	bool	false	false	Whether to enable TLS for the Redis connection.
redis_tls_ca_file	string		false	The path to the CA file for Redis TLS connection.
redis_tls_cert_file	string		false	The path to the client certificate file for Redis TLS connection.
redis_tls_key_file	string		false	The path to the client key file for Redis TLS connection.

Example Configuration

Component Telemetry

This component emits telemetry that can provide insight into how it is performing. The collector is configured to emit these metrics to localhost:8888/metrics by default.

Metric Name	Type	Description
otelcol_s3event_batch_size	histogram	The number of logs in a batch.
otelcol_s3event_objects_handled_total	counter	The number of S3 objects processed by the receiver.
otelcol_s3event_failures_total	counter	The number of failures encountered while processing S3 objects.