# Grafana Tempo

### Supported Telemetry Types

| Telemetry Type | Supported |
| -------------- | --------- |
| Traces         | ✓         |
| Metrics        |           |
| Logs           |           |

### Prerequisites

Before configuring the Grafana Tempo destination, ensure you have:

* A running Grafana Tempo instance (local or remote)
* Network connectivity from your Bindplane agent to the Tempo instance
* The OTLP receiver enabled and configured in your Tempo deployment
* The appropriate network ports open (default: gRPC `4317` or HTTP `4318`)
* Optional: Basic authentication credentials (if enabled on Tempo)
* Optional: TLS certificates (if using HTTPS/mTLS)

For help deploying Grafana Tempo, see the [Grafana Tempo deployment documentation](https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/).

### Configuration

#### Basic Configuration

The minimum configuration requires only the hostname and protocol. The destination will use gRPC protocol on port `4317` by default.

**Parameters:**

| Parameter   | Type    | Default | Description                                           | Required                                           |
| ----------- | ------- | ------- | ----------------------------------------------------- | -------------------------------------------------- |
| Hostname    | String  | -       | Hostname or IP address where Grafana Tempo is running | Yes                                                |
| Protocol    | Enum    | `grpc`  | The OTLP protocol to use (`grpc` or `http`)           | Yes                                                |
| Port (gRPC) | Integer | `4317`  | Port for gRPC protocol                                | Required if the corresponding protocol is selected |
| Port (HTTP) | Integer | `4318`  | Port for HTTP protocol                                | Required if the corresponding protocol is selected |

### Advanced Configuration

#### Authentication

Basic authentication can be enabled for Grafana Tempo endpoints that require it.

| Parameter      | Type    | Default | Description                       | Required                                |
| -------------- | ------- | ------- | --------------------------------- | --------------------------------------- |
| Use Basic Auth | Boolean | `false` | Enable basic authentication       | No                                      |
| Username       | String  | -       | Username for basic authentication | Required if `Use Basic Auth` is enabled |
| Password       | String  | -       | Password for basic authentication | Required if `Use Basic Auth` is enabled |

#### TLS Configuration

TLS encryption is available for secure communication with Grafana Tempo.

| Parameter                         | Type    | Default | Description                       | Required                            |
| --------------------------------- | ------- | ------- | --------------------------------- | ----------------------------------- |
| Enable TLS                        | Boolean | `false` | Enable TLS encryption             | No                                  |
| Skip TLS Certificate Verification | Boolean | `false` | Skip TLS certificate verification | No                                  |
| TLS Certificate Authority File    | String  | -       | Path to CA certificate file       | No                                  |
| Mutual TLS                        | Boolean | `false` | Enable mutual TLS (mTLS)          | No                                  |
| TLS Client Certificate File       | String  | -       | Path to client certificate file   | Required if `Mutual TLS` is enabled |
| TLS Client Private Key File       | String  | -       | Path to client private key file   | Required if `Mutual TLS` is enabled |

#### Compression

Compression reduces bandwidth usage when sending traces to Grafana Tempo.

| Parameter          | Type | Default | Description                                                                | Protocol |
| ------------------ | ---- | ------- | -------------------------------------------------------------------------- | -------- |
| Compression (gRPC) | Enum | `gzip`  | Compression for gRPC (`none`, `gzip`, `snappy`, `zstd`)                    | gRPC     |
| Compression (HTTP) | Enum | `gzip`  | Compression for HTTP (`none`, `gzip`, `deflate`, `snappy`, `zlib`, `zstd`) | HTTP     |

#### Additional Options

| Parameter          | Type    | Default | Description                                        |
| ------------------ | ------- | ------- | -------------------------------------------------- |
| Additional Headers | Map     | `{}`    | Additional HTTP headers to attach to requests      |
| HTTP Path Prefix   | String  | -       | Optional path prefix for HTTP endpoint (HTTP only) |
| Timeout            | Integer | `30`    | Timeout in seconds for sending batches             |

#### Retry and Queuing

This destination supports the [retry settings](/configuration/bindplane-otel-collector/retry-on-failure.md), the [sending queue settings](/configuration/bindplane-otel-collector/sending-queue.md), and the [persistent queue settings.](/configuration/bindplane-otel-collector/persistent-queue.md)

| Sending Queue | Persistent Queue | Retry on Failure |
| ------------- | ---------------- | ---------------- |
| ✓             | ✓                | ✓                |

### Examples

#### Basic gRPC Configuration

This example configures the destination to send traces to a local Grafana Tempo instance using gRPC.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: grafana-tempo
  name: Grafana Tempo
spec:
  type: grafana_tempo_otlp
  parameters:
    - name: telemetry_types
      value: ['Traces']
    - name: hostname
      value: 'localhost'
    - name: protocol
      value: 'grpc'
    - name: grpc_port
      value: 4317
```

#### HTTP Configuration with Compression

This example uses the HTTP protocol with gzip compression.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: grafana-tempo-http
  name: Grafana Tempo (HTTP)
spec:
  type: grafana_tempo_otlp
  parameters:
    - name: telemetry_types
      value: ['Traces']
    - name: hostname
      value: 'tempo.example.com'
    - name: protocol
      value: 'http'
    - name: http_port
      value: 4318
    - name: http_compression
      value: 'gzip'
```

#### Secure Configuration with TLS and Basic Authentication

This example demonstrates a production-ready configuration with TLS encryption and basic authentication.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: grafana-tempo-secure
  name: Grafana Tempo (Secure)
spec:
  type: grafana_tempo_otlp
  parameters:
    - name: telemetry_types
      value: ['Traces']
    - name: hostname
      value: 'tempo.example.com'
    - name: protocol
      value: 'grpc'
    - name: grpc_port
      value: 4317
    - name: enable_tls
      value: true
    - name: insecure_skip_verify
      value: false
    - name: ca_file
      value: '/etc/ssl/certs/ca.crt'
    - name: use_basic_auth
      value: true
    - name: basic_auth_username
      value: 'tempo-user'
    - name: basic_auth_password
      value: 'secure-password'
    - name: grpc_compression
      value: 'gzip'
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: true
    - name: persistent_queue_storage_extension
      value: 
        type: file_storage_persistent_queue
        parameters:
          - name: persistent_queue_directory
            value: ${OIQ_OTEL_COLLECTOR_HOME}/storage
          - name: timeout
            value: 1
          - name: persistent_queue_fsync
            value: true
```

#### Mutual TLS Configuration

This example shows how to configure mutual TLS (mTLS) with client certificates.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: grafana-tempo-mtls
  name: Grafana Tempo (mTLS)
spec:
  type: grafana_tempo_otlp
  parameters:
    - name: telemetry_types
      value: ['Traces']
    - name: hostname
      value: 'tempo.example.com'
    - name: protocol
      value: 'grpc'
    - name: grpc_port
      value: 4317
    - name: enable_tls
      value: true
    - name: insecure_skip_verify
      value: false
    - name: ca_file
      value: '/etc/ssl/certs/ca.crt'
    - name: mutual_tls
      value: true
    - name: cert_file
      value: '/etc/ssl/certs/client.crt'
    - name: key_file
      value: '/etc/ssl/private/client.key'
    - name: grpc_compression
      value: 'gzip'
```

### Configuration Tips

#### Protocol Selection

* **gRPC**: Recommended for most use cases. Binary protocol with lower bandwidth requirements and efficient compression.
* **HTTP**: More compatible with proxies and firewalls. Use this if you're behind a corporate proxy that only allows HTTP traffic.

#### Compression Recommendations

* **gzip**: Default choice. Good balance between compression ratio and CPU usage.
* **snappy**: Fast compression with lower CPU overhead. Good for high-throughput scenarios.
* **zstd**: Best compression ratio but requires higher CPU. Use for bandwidth-constrained environments.
* **none**: Disable compression if network throughput isn't a concern or if you want to reduce CPU usage.

#### Security Best Practices

* Always enable TLS in production environments.
* Set Skip TLS Certificate Verification to `false` and provide a CA certificate when possible.
* Use basic authentication only over TLS connections.
* Consider mutual TLS for highly sensitive environments or zero-trust networks.
* Store credentials securely using environment variables or secret management systems.
* Regularly rotate certificates and credentials.

#### Performance Tuning

* Increase Queue Size for high-volume trace collection (default: 5000).
* Adjust Number of Consumers based on available CPU and network bandwidth.
* Enable Persistent Queuing for mission-critical trace collection.
* Monitor queue depth and adjust settings if queues are consistently full.
* For network-constrained environments, consider enabling compression and persistent queuing.

#### Network Considerations

* Ensure network connectivity between your Bindplane agent and the Grafana Tempo endpoint.
* Open firewall rules for the appropriate port (`4317` for gRPC, `4318` for HTTP).
* If behind a corporate firewall, verify that HTTP/HTTPS is allowed for the selected protocol.
* Test connectivity before deployment: `telnet <hostname> <port>`

### Troubleshooting

#### Connection Refused

**Symptoms**: Traces are not being sent; connection timeout errors in logs.

**Solutions**:

1. Verify Grafana Tempo is running: `docker ps` or check systemd status
2. Verify the hostname/IP address is correct
3. Verify the port is correct (default: `4317` for gRPC, `4318` for HTTP)
4. Test network connectivity: `telnet <hostname> <port>`
5. Check firewall rules allow traffic on the port

#### TLS Certificate Errors

**Symptoms**: "certificate verify failed" or "bad certificate" errors in logs.

**Solutions**:

1. Verify the CA certificate path exists and is readable
2. Check certificate validity: `openssl x509 -in <cert> -noout -dates`
3. Verify the certificate matches the hostname used in configuration
4. For self-signed certificates, ensure the CA certificate is installed correctly
5. Temporarily set Skip TLS Certificate Verification to `true` for testing (not recommended for production)

#### Authentication Failures

**Symptoms**: "401 Unauthorized" or authentication errors in logs.

**Solutions**:

1. Verify username and password are correct
2. Check that basic authentication is enabled on the Grafana Tempo OTLP receiver
3. Ensure credentials are being transmitted over TLS (never send credentials over HTTP)
4. Verify credentials don't contain special characters that need escaping in YAML

#### High Queue Depth

**Symptoms**: Sending queue is consistently full; traces may be dropped if persistent queuing is disabled.

**Solutions**:

1. Verify Grafana Tempo instance can handle the trace volume
2. Check for network latency or bandwidth issues
3. Increase Queue Size to buffer more spans
4. Increase Number of Consumers to process traces faster
5. Enable Persistent Queuing to prevent trace loss during outages
6. Consider distributing traces across multiple Tempo instances

#### Data Loss

**Symptoms**: Traces are dropped after restarts or network outages.

**Solutions**:

1. Enable Persistent Queuing
2. Verify persistent queue directory exists and has write permissions
3. Monitor queue size to ensure it's not overflowing
4. Enable Retry on Failure
5. Check available disk space for persistent queue storage
6. Review logs for exporter errors or capacity issues

### Related Resources

* [Grafana Tempo Documentation](https://grafana.com/docs/tempo/latest/)
* [Grafana Tempo OTLP Receiver Configuration](https://grafana.com/docs/tempo/latest/configuration/#otlp-receiver)
* [OpenTelemetry Protocol Specification](https://opentelemetry.io/docs/specs/otel/protocol/)
* [Retry and Queuing Configuration](https://docs.bindplane.com/configuration/bindplane-otel-collector/retry-and-queueing)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.bindplane.com/integrations/destinations/grafana-tempo.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.