# Grafana Loki

### Supported Telemetry Types

| Telemetry Type | Supported |
| -------------- | --------- |
| Traces         |           |
| Metrics        |           |
| Logs           | ✓         |

### Prerequisites

Before configuring the Grafana Loki destination, ensure you have:

* A running Grafana Loki instance (local or remote)
* Network connectivity from your Bindplane agent to the Loki instance
* The OTLP HTTP receiver enabled and configured in your Loki deployment
* The appropriate network ports open (default: HTTP `3100`)
* Optional: Basic authentication credentials (if enabled on Loki)
* Optional: TLS certificates (if using HTTPS/mTLS)

For help deploying Grafana Loki, see the [Grafana Loki deployment documentation](https://grafana.com/docs/loki/latest/setup/) and [OTLP configuration guide](https://grafana.com/docs/loki/latest/send-data/otel/).

### Configuration

#### Basic Configuration

The minimum configuration requires only the Loki endpoint. The endpoint should point to your Loki OTLP HTTP receiver.

**Parameters:**

| Parameter | Type   | Default | Description                                                 | Required |
| --------- | ------ | ------- | ----------------------------------------------------------- | -------- |
| Endpoint  | String | -       | Loki ingestion address (e.g., <http://localhost:3100/otlp>) | Yes      |

***

### Advanced Configuration

#### Authentication

Basic authentication can be enabled for Grafana Loki endpoints that require it.

| Parameter      | Type    | Default | Description                       | Required                                |
| -------------- | ------- | ------- | --------------------------------- | --------------------------------------- |
| Use Basic Auth | Boolean | `false` | Enable basic authentication       | No                                      |
| Username       | String  | -       | Username for basic authentication | Required if `Use Basic Auth` is enabled |
| Password       | String  | -       | Password for basic authentication | Required if `Use Basic Auth` is enabled |

#### General Settings

Settings for controlling log processing and timeout behavior.

| Parameter     | Type    | Default | Description                                                         | Required |
| ------------- | ------- | ------- | ------------------------------------------------------------------- | -------- |
| Drop Raw Copy | Boolean | `true`  | Drop the raw copy of the log record stored in `log.record.original` | No       |
| Timeout       | Integer | `30`    | Timeout in seconds for sending batches to Loki                      | Yes      |

#### TLS Configuration

TLS encryption is available for secure communication with Grafana Loki.

| Parameter                         | Type    | Default | Description                       | Required                            |
| --------------------------------- | ------- | ------- | --------------------------------- | ----------------------------------- |
| Enable TLS                        | Boolean | `false` | Enable TLS encryption             | No                                  |
| Skip TLS Certificate Verification | Boolean | `false` | Skip TLS certificate verification | No                                  |
| TLS Certificate Authority File    | String  | -       | Path to CA certificate file       | No                                  |
| Mutual TLS                        | Boolean | `false` | Enable mutual TLS (mTLS)          | No                                  |
| TLS Client Certificate File       | String  | -       | Path to client certificate file   | Required if `Mutual TLS` is enabled |
| TLS Client Private Key File       | String  | -       | Path to client private key file   | Required if `Mutual TLS` is enabled |

#### Retry and Queuing

This destination supports the [retry settings](/configuration/bindplane-otel-collector/retry-on-failure.md), the [sending queue settings](/configuration/bindplane-otel-collector/sending-queue.md), and the [persistent queue settings.](/configuration/bindplane-otel-collector/persistent-queue.md)

| Sending Queue | Persistent Queue | Retry on Failure |
| ------------- | ---------------- | ---------------- |
| ✓             | ✓                | ✓                |

### Examples

#### Basic Configuration (Local Loki)

This example configures the destination to send logs to a local Grafana Loki instance.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: loki
  name: Grafana Loki
spec:
  type: loki_otlp
  parameters:
    - name: endpoint
      value: 'http://localhost:3100/otlp'
    - name: drop_raw_copy
      value: true
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: false
```

#### Configuration with Basic Authentication

This example demonstrates a production-ready configuration with basic authentication.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: loki-authenticated
  name: Grafana Loki (Authenticated)
spec:
  type: loki_otlp
  parameters:
    - name: endpoint
      value: 'https://loki.example.com/otlp'
    - name: use_basic_auth
      value: true
    - name: basic_auth_username
      value: 'loki-user'
    - name: basic_auth_password
      value: 'secure-password'
    - name: drop_raw_copy
      value: true
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: true
    - name: persistent_queue_storage_extension
      value: 
        type: file_storage_persistent_queue
        parameters:
          - name: persistent_queue_directory
            value: ${OIQ_OTEL_COLLECTOR_HOME}/storage
          - name: timeout
            value: 1
          - name: persistent_queue_fsync
            value: true
```

#### Secure Configuration with TLS and Basic Authentication

This example demonstrates a highly secure production configuration with TLS encryption and authentication.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: loki-secure
  name: Grafana Loki (Secure)
spec:
  type: loki_otlp
  parameters:
    - name: endpoint
      value: 'https://loki.example.com/otlp'
    - name: use_basic_auth
      value: true
    - name: basic_auth_username
      value: 'loki-user'
    - name: basic_auth_password
      value: 'secure-password'
    - name: configure_tls
      value: true
    - name: insecure_skip_verify
      value: false
    - name: ca_file
      value: '/etc/ssl/certs/ca.crt'
    - name: drop_raw_copy
      value: true
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: true
    - name: persistent_queue_storage_extension
      value: 
        type: file_storage_persistent_queue
        parameters:
          - name: persistent_queue_directory
            value: ${OIQ_OTEL_COLLECTOR_HOME}/storage
          - name: timeout
            value: 1
          - name: persistent_queue_fsync
            value: true
```

#### Mutual TLS Configuration

This example shows how to configure mutual TLS (mTLS) with client certificates.

```yaml
apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: loki-mtls
  name: Grafana Loki (mTLS)
spec:
  type: loki_otlp
  parameters:
    - name: endpoint
      value: 'https://loki.example.com/otlp'
    - name: configure_tls
      value: true
    - name: insecure_skip_verify
      value: false
    - name: ca_file
      value: '/etc/ssl/certs/ca.crt'
    - name: mutual_tls
      value: true
    - name: cert_file
      value: '/etc/ssl/certs/client.crt'
    - name: key_file
      value: '/etc/ssl/private/client.key'
    - name: drop_raw_copy
      value: true
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: true
    - name: persistent_queue_storage_extension
      value: 
        type: file_storage_persistent_queue
        parameters:
          - name: persistent_queue_directory
            value: ${OIQ_OTEL_COLLECTOR_HOME}/storage
          - name: timeout
            value: 1
          - name: persistent_queue_fsync
            value: true
```

### Configuration Tips

#### Endpoint Configuration

* Use the format: `http://hostname:port/otlp` or `https://hostname:port/otlp`
* Default port for Loki OTLP is `3100`
* The `/otlp` path is required in the endpoint URL
* Verify Loki is configured to accept OTLP logs
* Test the endpoint: `curl -I http://localhost:3100/otlp`

#### Raw Copy Handling

* Enable **Drop Raw Copy** to reduce storage by removing the `log.record.original` attribute
* Disable only if you need the original raw log record for debugging
* Default behavior (enabled) is recommended for production environments to save storage space

#### Security Best Practices

* Always use HTTPS in production environments
* Set Skip TLS Certificate Verification to `false` and provide a CA certificate when possible
* Use basic authentication only over HTTPS/TLS connections
* Consider mutual TLS for highly sensitive environments or zero-trust networks
* Store credentials securely using environment variables or secret management systems
* Regularly rotate credentials and certificates
* Use Grafana service accounts when available for better access control

#### Performance Tuning

* Increase Queue Size for high-volume log collection (default: `5000`)
* Adjust Number of Consumers based on available CPU and network bandwidth
* Enable Persistent Queuing for mission-critical log collection
* Monitor queue depth and adjust settings if queues are consistently full
* Enable Synchronize Persistent Queue to Disk in critical environments (with performance trade-off)
* Disable Synchronize Persistent Queue to Disk for better performance when acceptable data loss risk is low

#### Network Considerations

* Ensure network connectivity between your Bindplane agent and the Grafana Loki endpoint
* Verify the OTLP HTTP receiver is enabled in your Loki configuration
* Open firewall rules for the appropriate port (default: `3100`)
* Check if a proxy is required and configure accordingly
* Test connectivity before deployment: `curl -I https://endpoint/otlp`

***

### Troubleshooting

#### Connection Refused

**Symptoms**: Logs are not being sent; connection timeout errors in logs.

**Solutions**:

1. Verify Grafana Loki is running: `docker ps` or check systemd status
2. Verify the endpoint URL is correct (should include `/otlp` path)
3. Test connectivity: `curl -I http://endpoint:3100/otlp`
4. Verify network firewall rules allow traffic on the port
5. Check Loki logs for errors

#### TLS Certificate Errors

**Symptoms**: "certificate verify failed" or "bad certificate" errors in logs.

**Solutions**:

1. Verify the CA certificate path exists and is readable
2. Check certificate validity: `openssl x509 -in <cert> -noout -dates`
3. Verify the certificate matches the hostname used in configuration
4. For self-signed certificates, ensure the CA certificate is installed correctly
5. Temporarily set Skip TLS Certificate Verification to `true` for testing (not recommended for production)

#### Authentication Failures

**Symptoms**: "401 Unauthorized" or authentication errors in logs.

**Solutions**:

1. Verify username and password are correct
2. Check that basic authentication is enabled on the Grafana Loki OTLP receiver
3. Ensure credentials are being transmitted over HTTPS/TLS
4. Verify credentials don't contain special characters that need escaping in YAML

#### Logs Not Appearing in Loki

**Symptoms**: Logs are sent but don't appear in Loki UI or queries.

**Solutions**:

1. Verify logs are being generated by the source
2. Check that Loki has at least one label configured (Loki requires labels for all logs)
3. Verify the OTLP receiver is configured correctly in Loki
4. Check Loki logs for ingestion errors or rejections
5. Verify log volume isn't exceeding Loki rate limits

#### High Queue Depth

**Symptoms**: Sending queue is consistently full; logs may be dropped if persistent queuing is disabled.

**Solutions**:

1. Verify Grafana Loki instance can handle the log volume
2. Check for network latency or bandwidth issues
3. Increase Queue Size to buffer more logs
4. Increase Number of Consumers to process batches faster
5. Enable Persistent Queuing to prevent log loss during outages
6. Consider distributing logs across multiple Loki instances

#### Performance Issues

**Symptoms**: High CPU or memory usage; slow log ingestion.

**Solutions**:

1. Monitor Bindplane agent CPU and memory usage
2. Consider reducing log volume at the source (sampling, filtering)
3. Disable Synchronize Persistent Queue to Disk if enabled (unless data integrity is critical)
4. Verify network connectivity and latency to Loki
5. Consider batching at the source to reduce log frequency

#### Data Loss

**Symptoms**: Logs are dropped after restarts or network outages.

**Solutions**:

1. Enable Persistent Queuing
2. Verify persistent queue directory exists and has write permissions
3. Monitor queue size to ensure it's not overflowing
4. Enable Retry on Failure to handle transient failures
5. Check available disk space for persistent queue storage
6. For critical environments, enable Synchronize Persistent Queue to Disk
7. Review logs for exporter errors or capacity issues

### Related Resources

* [Grafana Loki Documentation](https://grafana.com/docs/loki/latest/)
* [Grafana Loki OTLP Configuration](https://grafana.com/docs/loki/latest/send-data/otel/)
* [Grafana Loki Setup and Installation](https://grafana.com/docs/loki/latest/setup/)
* [OpenTelemetry Protocol Specification](https://opentelemetry.io/docs/specs/otel/protocol/)
* [Bindplane Destinations Overview](https://docs.bindplane.com/resources/destinations/)
* [Retry and Queuing Configuration](https://docs.bindplane.com/configuration/bindplane-otel-collector/retry-and-queueing)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.bindplane.com/integrations/destinations/loki.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.