Grafana Tempo

Send traces to Grafana Tempo using the OpenTelemetry Protocol (OTLP). Grafana Tempo is a high-scale distributed tracing backend that is easy to operate and cost-effective.

Supported Telemetry Types

Telemetry Type	Supported
Traces	✓
Metrics	✗
Logs	✗

Prerequisites

Before configuring the Grafana Tempo (OTLP) destination, ensure you have:

• A running Grafana Tempo instance (local or remote)

• Network connectivity from your Bindplane agent to the Tempo instance

• The OTLP receiver enabled and configured in your Tempo deployment

• The appropriate network ports open (default: gRPC 4317 or HTTP 4318)

• Optional: Basic authentication credentials (if enabled on Tempo)

• Optional: TLS certificates (if using HTTPS/mTLS)

For help deploying Grafana Tempo, see the Grafana Tempo deployment documentation.

Configuration

Basic Configuration

The minimum configuration requires only the hostname and protocol. The destination will use gRPC protocol on port 4317 by default.

Parameters:

Parameter	Type	Default	Description	Required
Hostname	String	-	Hostname or IP address where Grafana Tempo is running	Yes
Protocol	Enum	grpc	The OTLP protocol to use (grpc or http)	Yes
Port (gRPC)	Integer	4317	Port for gRPC protocol	Conditional*
Port (HTTP)	Integer	4318	Port for HTTP protocol	Conditional*

*Required if the corresponding protocol is selected

Advanced Configuration

Authentication

Basic authentication can be enabled for Grafana Tempo endpoints that require it.

Parameter	Type	Default	Description	Required
Use Basic Auth	Boolean	false	Enable basic authentication	No
Username	String	-	Username for basic authentication	Conditional*
Password	String	-	Password for basic authentication	Conditional*

*Required if Use Basic Auth is enabled

TLS Configuration

TLS encryption is available for secure communication with Grafana Tempo.

Parameter	Type	Default	Description	Required
Enable TLS	Boolean	false	Enable TLS encryption	No
Skip TLS Certificate Verification	Boolean	false	Skip TLS certificate verification	No
TLS Certificate Authority File	String	-	Path to CA certificate file	No
Mutual TLS	Boolean	false	Enable mutual TLS (mTLS)	No
TLS Client Certificate File	String	-	Path to client certificate file	Conditional**
TLS Client Private Key File	String	-	Path to client private key file	Conditional**

**Required if Mutual TLS is enabled

Compression

Compression reduces bandwidth usage when sending traces to Grafana Tempo.

Parameter	Type	Default	Description	Protocol
Compression (gRPC)	Enum	gzip	Compression for gRPC (none, gzip, snappy, zstd)	gRPC
Compression (HTTP)	Enum	gzip	Compression for HTTP (none, gzip, deflate, snappy, zlib, zstd)	HTTP

Additional Options

Parameter	Type	Default	Description
Additional Headers	Map	{}	Additional HTTP headers to attach to requests
HTTP Path Prefix	String	-	Optional path prefix for HTTP endpoint (HTTP only)
Timeout	Integer	30	Timeout in seconds for sending batches

Retry and Queuing

This destination supports the following retry and queuing settings:

Parameter	Type	Default	Description
Enable Retry on Failure	Boolean	true	Enable retry on failure
Initial Interval	Integer	5	Initial retry interval in seconds
Max Interval	Integer	30	Maximum retry interval in seconds
Max Elapsed Time	Integer	300	Maximum total retry time in seconds
Enable Sending Queue	Boolean	true	Enable sending queue for buffering
Number of Consumers	Integer	10	Number of queue consumers
Queue Size	Integer	5000	Maximum queue size in spans
Enable Persistent Queuing	Boolean	true	Enable persistent disk-based queuing
Persistent Queue Directory	String	${OIQ_OTEL_COLLECTOR_HOME}/storage	Directory for persistent queue storage

For more information, see Retry and Queuing Configuration.

Examples

Basic gRPC Configuration

This example configures the destination to send traces to a local Grafana Tempo instance using gRPC.

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: grafana-tempo
  name: Grafana Tempo
spec:
  type: grafana_tempo_otlp
  parameters:
    - name: telemetry_types
      value: ['Traces']
    - name: hostname
      value: 'localhost'
    - name: protocol
      value: 'grpc'
    - name: grpc_port
      value: 4317

HTTP Configuration with Compression

This example uses the HTTP protocol with gzip compression.

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: grafana-tempo-http
  name: Grafana Tempo (HTTP)
spec:
  type: grafana_tempo_otlp
  parameters:
    - name: telemetry_types
      value: ['Traces']
    - name: hostname
      value: 'tempo.example.com'
    - name: protocol
      value: 'http'
    - name: http_port
      value: 4318
    - name: http_compression
      value: 'gzip'

Secure Configuration with TLS and Basic Authentication

This example demonstrates a production-ready configuration with TLS encryption and basic authentication.

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: grafana-tempo-secure
  name: Grafana Tempo (Secure)
spec:
  type: grafana_tempo_otlp
  parameters:
    - name: telemetry_types
      value: ['Traces']
    - name: hostname
      value: 'tempo.example.com'
    - name: protocol
      value: 'grpc'
    - name: grpc_port
      value: 4317
    - name: enable_tls
      value: true
    - name: insecure_skip_verify
      value: false
    - name: ca_file
      value: '/etc/ssl/certs/ca.crt'
    - name: use_basic_auth
      value: true
    - name: basic_auth_username
      value: 'tempo-user'
    - name: basic_auth_password
      value: 'secure-password'
    - name: grpc_compression
      value: 'gzip'
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: true

Mutual TLS Configuration

This example shows how to configure mutual TLS (mTLS) with client certificates.

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: grafana-tempo-mtls
  name: Grafana Tempo (mTLS)
spec:
  type: grafana_tempo_otlp
  parameters:
    - name: telemetry_types
      value: ['Traces']
    - name: hostname
      value: 'tempo.example.com'
    - name: protocol
      value: 'grpc'
    - name: grpc_port
      value: 4317
    - name: enable_tls
      value: true
    - name: insecure_skip_verify
      value: false
    - name: ca_file
      value: '/etc/ssl/certs/ca.crt'
    - name: mutual_tls
      value: true
    - name: cert_file
      value: '/etc/ssl/certs/client.crt'
    - name: key_file
      value: '/etc/ssl/private/client.key'
    - name: grpc_compression
      value: 'gzip'

Configuration Tips

Protocol Selection

• gRPC: Recommended for most use cases. Binary protocol with lower bandwidth requirements and efficient compression.

• HTTP: More compatible with proxies and firewalls. Use this if you're behind a corporate proxy that only allows HTTP traffic.

Compression Recommendations

• gzip: Default choice. Good balance between compression ratio and CPU usage.

• snappy: Fast compression with lower CPU overhead. Good for high-throughput scenarios.

• zstd: Best compression ratio but requires higher CPU. Use for bandwidth-constrained environments.

• none: Disable compression if network throughput isn't a concern or if you want to reduce CPU usage.

Security Best Practices

• Always enable TLS in production environments.

• Set Skip TLS Certificate Verification to false and provide a CA certificate when possible.

• Use basic authentication only over TLS connections.

• Consider mutual TLS for highly sensitive environments or zero-trust networks.

• Store credentials securely using environment variables or secret management systems.

• Regularly rotate certificates and credentials.

Performance Tuning

• Increase Queue Size for high-volume trace collection (default: 5000).

• Adjust Number of Consumers based on available CPU and network bandwidth.

• Enable Persistent Queuing for mission-critical trace collection.

• Monitor queue depth and adjust settings if queues are consistently full.

• For network-constrained environments, consider enabling compression and persistent queuing.

Network Considerations

• Ensure network connectivity between your Bindplane agent and the Grafana Tempo endpoint.

• Open firewall rules for the appropriate port (4317 for gRPC, 4318 for HTTP).

• If behind a corporate firewall, verify that HTTP/HTTPS is allowed for the selected protocol.

• Test connectivity before deployment: telnet <hostname> <port>

Troubleshooting

Connection Refused

Symptoms: Traces are not being sent; connection timeout errors in logs.

Solutions:

1. Verify Grafana Tempo is running: docker ps or check systemd status

2. Verify the hostname/IP address is correct

3. Verify the port is correct (default: 4317 for gRPC, 4318 for HTTP)

4. Test network connectivity: telnet <hostname> <port>

5. Check firewall rules allow traffic on the port

TLS Certificate Errors

Symptoms: "certificate verify failed" or "bad certificate" errors in logs.

Solutions:

1. Verify the CA certificate path exists and is readable

2. Check certificate validity: openssl x509 -in <cert> -noout -dates

3. Verify the certificate matches the hostname used in configuration

4. For self-signed certificates, ensure the CA certificate is installed correctly

5. Temporarily set Skip TLS Certificate Verification to true for testing (not recommended for production)

Authentication Failures

Symptoms: "401 Unauthorized" or authentication errors in logs.

Solutions:

1. Verify username and password are correct

2. Check that basic authentication is enabled on the Grafana Tempo OTLP receiver

3. Ensure credentials are being transmitted over TLS (never send credentials over HTTP)

4. Verify credentials don't contain special characters that need escaping in YAML

High Queue Depth

Symptoms: Sending queue is consistently full; traces may be dropped if persistent queuing is disabled.

Solutions:

1. Verify Grafana Tempo instance can handle the trace volume

2. Check for network latency or bandwidth issues

3. Increase Queue Size to buffer more spans

4. Increase Number of Consumers to process traces faster

5. Enable Persistent Queuing to prevent trace loss during outages

6. Consider distributing traces across multiple Tempo instances

Data Loss

Symptoms: Traces are dropped after restarts or network outages.

Solutions:

1. Enable Persistent Queuing

2. Verify persistent queue directory exists and has write permissions

3. Monitor queue size to ensure it's not overflowing

4. Enable Retry on Failure

5. Check available disk space for persistent queue storage

6. Review logs for exporter errors or capacity issues

Related Resources

• Grafana Tempo Documentation

• Grafana Tempo OTLP Receiver Configuration

• OpenTelemetry Protocol Specification

• Retry and Queuing Configuration