Honeycomb Enhance Indexing S3
Export logs and traces to AWS S3 with automatic field indexing for efficient rehydration and querying in Honeycomb.
This is only for the Honeycomb License Plan
Supported Telemetry Types
Traces
✓
Metrics
Logs
✓
Prerequisites
A Honeycomb account with a Management API key and secret (must have
enhance:writescope)An AWS S3 bucket with appropriate write permissions
AWS credentials configured on the collector host
Network connectivity from the collector to both S3 and the Honeycomb API endpoint
Configuration
Basic Configuration
The minimum configuration requires Honeycomb API credentials, an S3 bucket, and an AWS region.
Telemetry Types
Selector
Logs, Traces
Select which types of telemetry to export.
Yes
API Key
String
Honeycomb Management API key with enhance:write scope.
Yes
API Secret
String
Honeycomb Management API secret.
Yes
API Endpoint
String
https://api.honeycomb.io
Honeycomb API endpoint URL.
Yes
Region
Enum
us-east-1
AWS region where the S3 bucket is located.
Yes
Bucket
String
Name of the S3 bucket to export telemetry into.
Yes
Folder Prefix
String
Root directory of the bucket to export telemetry into.
No
Partition Format
String
year=%Y/month=%m/day=%d/hour=%H/minute=%M
Time-based partition format using ctime format codes. Must contain year, month, day, hour, and minute placeholders.
Yes
Advanced Configuration
Data Format and Compression
Data Format
Enum
otlp_proto
Format of telemetry data written to S3. Options: otlp_proto (smaller files, more efficient) or otlp_json (human-readable).
No
Compression
Enum
gzip
Compression applied to data stored in S3. Options: gzip or none.
No
AWS Credentials
Role ARN
String
IAM Role ARN to assume for S3 access. If not set, the collector uses the default AWS SDK credential chain (environment variables, IAM instance profile, shared credentials file).
No
Note: This destination does not accept inline AWS access keys or secret keys. The collector resolves credentials using the standard AWS SDK credential chain. You can optionally specify a Role ARN to assume a cross-account or scoped IAM role. For all other credential methods (environment variables, instance profiles, shared credentials files), configure them on the collector host directly.
S3 Connection Options
Endpoint
String
Overrides the S3 endpoint URL. Use this for S3-compatible services like MinIO.
No
Force Path Style
Boolean
false
Force path-style S3 addressing instead of virtual-host style. Required for some S3-compatible services.
No
Disable SSL
Boolean
false
Disable SSL verification for S3 connections.
No
Indexing
Custom Indexed Fields
Strings
Additional fields to index beyond the automatic ones (trace.trace_id, service.name, session.id).
No
Note: To minimize Collector processing compute and time, Honeycomb recommends indexing no more than 5 custom fields.
General Settings
Drop Raw Copy
Boolean
true
Drop the raw copy of the log record stored in log.record.original.
No
Timeout
Integer
30
Maximum time in seconds to wait for a batch to be sent.
No
Retry Mode
Enum
standard
S3 client retry strategy. Options: standard or adaptive.
No
Maximum Retries
Integer
3
Maximum number of retry attempts for S3 uploads. Set to 0 for unlimited retries.
No
Retry and Queuing
This destination supports the retry settings, the sending queue settings, and the persistent queue settings.
Configuration Tips
S3 Bucket and Credentials
AWS credentials are resolved via the standard AWS SDK credential chain (IAM role, environment variables, or shared credentials file). This destination does not accept inline AWS access keys.
To use a specific IAM role (e.g., for cross-account access), set the Role ARN field in Advanced Configuration. The collector will assume this role using STS.
Make sure the S3 bucket policy grants
s3:PutObjectpermission to the collector's identity (or the assumed role).Use the Folder Prefix to isolate telemetry data within a shared bucket.
Custom Indexed Fields
Fields like
user.idorenvironmentcan be added to Custom Indexed Fields for faster rehydration queries in Honeycomb.Each additional indexed field increases S3 storage usage due to extra index files written alongside data files.
Honeycomb API Endpoint
Use
https://api.honeycomb.iofor US-based Honeycomb accounts.Use
https://api.eu1.honeycomb.iofor EU-based Honeycomb accounts.
Troubleshooting
S3 Upload Failures
Errors referencing AccessDenied, NoSuchBucket, or InvalidAccessKeyId in collector logs.
Verify the S3 bucket exists in the configured region and the collector host has valid AWS credentials with
s3:PutObjectpermission.If using a custom Endpoint for S3-compatible storage, make sure Force Path Style is enabled.
Honeycomb API Authentication Errors
Errors referencing 401 Unauthorized or 403 Forbidden from the Honeycomb API endpoint.
Confirm the API Key and API Secret are for a Management API key with the
enhance:writescope. Standard ingest keys will not work.Verify the API Endpoint matches your Honeycomb account region (US vs EU).
Missing or Incomplete Index Files
Data appears in S3 but index files are missing or incomplete. Rehydration queries in Honeycomb return fewer results than expected.
Check that the partition format contains all required placeholders (
%Y,%m,%d,%H,%M). Missing placeholders will cause validation failures.Verify that custom indexed field names match the exact attribute keys present in your telemetry data.
Related Resources
Last updated
Was this helpful?