Chronosphere

Send your logs, metrics, and traces to Chronosphere using the OpenTelemetry Protocol (OTLP).

Overview

Chronosphere is a cloud-native observability platform designed for microservices and container environments. It provides comprehensive monitoring, alerting, and visualization capabilities for metrics, logs, and distributed traces.

The Chronosphere OTLP destination allows you to export telemetry data from BindPlane directly to Chronosphere's Observability Platform using industry-standard OpenTelemetry protocols. This destination supports:

• Logs: Send structured logs with automatic processing and raw log handling options

• Metrics: Export metrics with validation response control and resource-to-telemetry conversion

• Traces: Collect and export distributed traces for performance analysis

Prerequisites

Before configuring the Chronosphere OTLP destination, you'll need:

1. Active Chronosphere Account: A Chronosphere Observability Platform account with access to your organization's instance

2. Tenant Name: Your organization's Chronosphere tenant name (the subdomain of your instance, e.g., if your URL is https://mycompany.chronosphere.io, your tenant name is mycompany)

3. API Token: A valid API token from a Chronosphere service account with write permissions for the telemetry types you plan to send

Creating an API Token in Chronosphere

To generate an API token:

1. Log in to your Chronosphere instance

2. Navigate to Administration → Accounts & Teams → Service Accounts

3. Click Create Service Account or select an existing account

4. Create or view an API token with the following scopes:

   • Write metrics (for metrics data)

   • Write logs (for logs data)

   • Write traces (for traces data)

5. Copy the generated token securely (you'll need this for the BindPlane configuration)

Configuration

Basic Connection Settings

To configure a basic Chronosphere OTLP destination:

1. In BindPlane, create a new destination

2. Select Chronosphere OTLP as the destination type

3. Enter your configuration:

Parameter	Description	Required
Tenant Name	Your Chronosphere organization name (subdomain)	Yes
API Token	Service account API token with write permissions	Yes
Protocol	HTTP/1.1 or HTTP/2 protocol version	No (default: HTTP/1.1)
Compression	Compression algorithm: none, gzip, or zstd	No (default: zstd)

Telemetry Type Selection

Choose which telemetry types you want to send:

• Logs: Enable to send structured log data to Chronosphere

• Metrics: Enable to send metric data for monitoring

• Traces: Enable to send distributed tracing data

You can enable any combination of these types.

Batching Configuration

Chronosphere recommends specific batch processor settings to prevent payload size violations:

Parameter	Recommended Value	Description
Batch Send Size	1000	Number of items to trigger batch sending
Batch Max Size	2000	Maximum items per batch before forcing split
Batch Timeout	1s	Maximum time to wait before sending partial batch

These defaults optimize data delivery while respecting Chronosphere's payload size limits:

• Compressed payloads must be smaller than 16 MB

• Metric payloads must contain fewer than 10,000 items

TLS & Security Configuration

Basic TLS (Certificate Verification)

For standard HTTPS connections with server certificate verification:

Parameter	Description
Enable TLS	Enable TLS encryption (required by Chronosphere)
CA Certificate File	Path to CA certificate file for server verification
Insecure Skip Verify	Skip server certificate verification (development only)

Mutual TLS (mTLS)

For mutual TLS authentication:

Parameter	Description
Enable TLS	Enable TLS encryption
CA Certificate File	Path to CA certificate for server verification
Client Certificate File	Path to client certificate file (mTLS)
Client Key File	Path to client private key file (mTLS)

Security Notes:

• Never use Insecure Skip Verify in production

• Store certificate files securely with appropriate permissions

• Regularly rotate certificates according to your security policy

• Consider using BindPlane's secret management for sensitive values

Advanced Configuration

HTTP Headers

Add custom HTTP headers to requests:

http_headers:
  X-Custom-Header: "value"
  Another-Header: "another-value"

Headers are merged with default headers (including the API-Token header).

Retry Configuration

Chronosphere recommends enabling retries to handle transient failures:

Parameter	Recommended Value	Description
Retry on Failure	Enabled	Automatically retry failed requests
Initial Interval	5s	Time to wait before first retry
Max Interval	30s	Maximum wait between retries
Max Elapsed Time	300s	Total time to spend retrying

Persistent Queuing

For high-volume scenarios or when reliability is critical:

Parameter	Description
Sending Queue Size	Number of items to buffer in memory
Persistent Queue Directory	Directory for disk-based storage (optional)
Persistent Queue Fsync	Force filesystem sync on each write (safety vs. performance)

When persistent queue is enabled, data is buffered to disk if the destination is unavailable, ensuring no data loss.

Metrics-Specific Options

For metrics telemetry:

Parameter	Options	Description
Metrics Validation Response	SHORT, SUMMARY, DETAILED	Verbosity of partial success responses
Resource to Telemetry Conversion	Enabled/Disabled	Convert resource attributes to metric attributes

Telemetry-Specific Processing

Logs Processing

When logs are enabled, the destination:

• Collects structured log records

• Removes the raw log copy attribute (log.record.original) to reduce payload size

• Maintains all other log attributes and fields

Metrics Processing

When metrics are enabled:

• All metric types (counters, gauges, histograms, exponential histograms) are supported

• Resource attributes can be converted to metric attributes based on configuration

• Partial success responses provide detailed validation feedback

• Resource-to-telemetry conversion helps normalize data across different sources

Traces Processing

When traces are enabled:

• Distributed trace spans are collected and exported

• All span attributes and resource information is preserved

• Trace context propagation is maintained for correlation

Configuration Examples

Minimal Configuration (Logs Only)

name: chronosphere-logs
type: chronosphere_otlp
parameters:
  tenant_name: mycompany
  api_token: ${CHRONOSPHERE_API_TOKEN}
  send_logs: true

Full Configuration (All Telemetry Types)

name: chronosphere-full
type: chronosphere_otlp
parameters:
  tenant_name: mycompany
  api_token: ${CHRONOSPHERE_API_TOKEN}
  send_logs: true
  send_metrics: true
  send_traces: true
  protocol: "http/1.1"
  compression: "zstd"
  batch_send_size: 1000
  batch_max_size: 2000
  batch_timeout: "1s"
  enable_tls: true
  retry_on_failure: true
  retry_initial_interval: "5s"
  retry_max_interval: "30s"
  retry_max_elapsed_time: "300s"

Mutual TLS Configuration

name: chronosphere-mtls
type: chronosphere_otlp
parameters:
  tenant_name: mycompany
  api_token: ${CHRONOSPHERE_API_TOKEN}
  send_logs: true
  send_metrics: true
  send_traces: true
  enable_tls: true
  ca_file: "/etc/ssl/certs/ca-bundle.crt"
  client_cert_file: "/etc/ssl/certs/client.crt"
  client_key_file: "/etc/ssl/private/client.key"
  mutual_tls: true
  retry_on_failure: true
  sending_queue_enabled: true
  sending_queue_num_consumers: 10
  sending_queue_queue_size: 5000
  persistent_queue_directory: "/var/lib/bindplane/queue"

High-Volume Optimization

For high-volume telemetry scenarios:

name: chronosphere-high-volume
type: chronosphere_otlp
parameters:
  tenant_name: mycompany
  api_token: ${CHRONOSPHERE_API_TOKEN}
  send_logs: true
  send_metrics: true
  send_traces: true
  protocol: "http/2"
  compression: "zstd"
  batch_send_size: 1500
  batch_max_size: 2000
  batch_timeout: "2s"
  timeout: "45s"
  sending_queue_enabled: true
  sending_queue_num_consumers: 20
  sending_queue_queue_size: 10000
  persistent_queue_directory: "/var/lib/bindplane/queue"
  persistent_queue_fsync: false
  retry_on_failure: true
  retry_initial_interval: "5s"
  retry_max_interval: "30s"
  retry_max_elapsed_time: "300s"

Performance Optimization

Batch Processing

The batch processor configuration significantly impacts performance:

• Batch Send Size: Set to 1000 items (Chronosphere recommendation)

• Batch Max Size: Set to 2000 items to prevent exceeding limits

• Batch Timeout: Set to 1 second for real-time responsiveness with some batching efficiency

Larger batch sizes reduce overhead but increase latency; smaller sizes provide faster delivery but with more requests.

Compression

Chronosphere supports three compression algorithms:

Algorithm	Pros	Cons
zstd	Better compression, faster decompression (recommended)	Higher CPU usage for compression
gzip	Good compression, wide support	Slower decompression
none	No compression overhead	Larger payloads, more bandwidth

Use zstd for most scenarios; disable compression only for low-volume or development environments.

Queue Configuration

For reliability and burst handling:

• Queue Size: Start with 5000-10000 for burst protection

• Num Consumers: Use 10-20 concurrent senders for high volume

• Persistent Directory: Enable for critical data paths

• Fsync: Disable for better performance (data still safe with reasonable queue depth)

Timeout Settings

Chronosphere recommends:

• Default Timeout: 30 seconds for most requests

• High-Latency Networks: 45-60 seconds

• HTTP/2: Can use same as HTTP/1.1

Troubleshooting

Authentication Errors

Error: "401 Unauthorized" or "Invalid API Token"

Solutions:

1. Verify your API token is correct and hasn't expired

2. Check that the service account has write permissions for your telemetry types

3. Ensure the API token is properly formatted (no extra whitespace)

4. Regenerate the token in Chronosphere if necessary

Connection Timeouts

Error: "Connection timeout" or "Request timeout"

Solutions:

1. Check network connectivity to TENANT.chronosphere.io:443

2. Increase the timeout setting from the default 30 seconds

3. Verify firewall rules and proxies aren't blocking the connection

4. Confirm your Chronosphere instance is operational

TLS/Certificate Errors

Error: "certificate verify failed" or "x509 certificate error"

Solutions:

1. Verify the CA certificate file path is correct and readable

2. Ensure the certificate is not expired

3. For self-signed certificates, use the correct CA certificate file

4. Check that the certificate CN or SAN matches the endpoint hostname

5. For mutual TLS, verify both client certificate and key files are valid

Data Not Arriving

Error: No data appears in Chronosphere despite successful sending

Solutions:

1. Verify data is being generated upstream (check source configuration)

2. Test with a simple configuration first (logs only)

3. Check Chronosphere's incoming data dashboard for validation errors

4. Review the Chronosphere API token scopes for your telemetry type

5. Verify the tenant name matches your Chronosphere instance

Partial Success Responses

Issue: Seeing "partial success" responses with rejected data

Troubleshooting:

1. Check the Metrics Validation Response setting to understand rejection reasons

2. Set to DETAILED for verbose error messages

3. Review rejected metrics for cardinality or formatting issues

4. Adjust cardinality reduction in upstream processors if needed

High CPU Usage

Issue: High CPU usage or slow performance

Solutions:

1. Reduce batch size to decrease CPU time per batch

2. Switch compression from zstd to gzip

3. Increase batch timeout to allow more items per batch

4. Disable persistent queue fsync if reliability margin allows

Related Resources

• Chronosphere OTLP Endpoints Documentation

• Chronosphere OTLP Logs Documentation

• Chronosphere Service Accounts

• OpenTelemetry Collector Documentation

• OTLP Protocol Specification

Support

For issues or questions:

1. Check the troubleshooting section above

2. Review your BindPlane logs for error details

3. Verify Chronosphere status at https://chronosphere.io/

4. Contact Chronosphere Support for platform-specific issues

5. Contact BindPlane Support for integration issues