Grafana Loki

Send logs to Grafana Loki using the OpenTelemetry Protocol (OTLP) HTTP exporter. Grafana Loki is a high-volume log aggregation system designed to be cost-effective and easy to operate.

Supported Telemetry Types

Telemetry Type	Supported
Traces	✗
Metrics	✗
Logs	✓

Prerequisites

Before configuring the Grafana Loki (OTLP) destination, ensure you have:

• A running Grafana Loki instance (local or remote)

• Network connectivity from your Bindplane agent to the Loki instance

• The OTLP HTTP receiver enabled and configured in your Loki deployment

• The appropriate network ports open (default: HTTP 3100)

• Optional: Basic authentication credentials (if enabled on Loki)

• Optional: TLS certificates (if using HTTPS/mTLS)

For help deploying Grafana Loki, see the Grafana Loki deployment documentation and OTLP configuration guide.

Configuration

Basic Configuration

The minimum configuration requires only the Loki endpoint. The endpoint should point to your Loki OTLP HTTP receiver.

Parameters:

Parameter	Type	Default	Description	Required
Endpoint	String	-	Loki ingestion address (e.g., http://localhost:3100/otlp)	Yes

---

Advanced Configuration

Authentication

Basic authentication can be enabled for Grafana Loki endpoints that require it.

Parameter	Type	Default	Description	Required
Use Basic Auth	Boolean	false	Enable basic authentication	No
Username	String	-	Username for basic authentication	Conditional*
Password	String	-	Password for basic authentication	Conditional*

*Required if Use Basic Auth is enabled

General Settings

Settings for controlling log processing and timeout behavior.

Parameter	Type	Default	Description	Required
Drop Raw Copy	Boolean	true	Drop the raw copy of the log record stored in log.record.original	No
Timeout	Integer	30	Timeout in seconds for sending batches to Loki	Yes

TLS Configuration

TLS encryption is available for secure communication with Grafana Loki.

Parameter	Type	Default	Description	Required
Enable TLS	Boolean	false	Enable TLS encryption	No
Skip TLS Certificate Verification	Boolean	false	Skip TLS certificate verification	No
TLS Certificate Authority File	String	-	Path to CA certificate file	No
Mutual TLS	Boolean	false	Enable mutual TLS (mTLS)	No
TLS Client Certificate File	String	-	Path to client certificate file	Conditional**
TLS Client Private Key File	String	-	Path to client private key file	Conditional**

**Required if Mutual TLS is enabled

Retry and Queuing

This destination supports the following retry and queuing settings:

Parameter	Type	Default	Description
Enable Retry on Failure	Boolean	true	Enable retry on failure
Initial Interval	Integer	5	Initial retry interval in seconds
Max Interval	Integer	30	Maximum retry interval in seconds
Max Elapsed Time	Integer	300	Maximum total retry time in seconds
Enable Sending Queue	Boolean	true	Enable sending queue for buffering
Number of Consumers	Integer	10	Number of queue consumers
Queue Size	Integer	5000	Maximum queue size in batches
Enable Persistent Queuing	Boolean	true	Enable persistent disk-based queuing
Persistent Queue Directory	String	${OIQ_OTEL_COLLECTOR_HOME}/storage	Directory for persistent queue storage
Synchronize Persistent Queue to Disk	Boolean	false	Synchronize persistent queue to disk for data integrity

For more information, see Retry and Queuing Configuration.

Examples

Basic Configuration (Local Loki)

This example configures the destination to send logs to a local Grafana Loki instance.

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: loki
  name: Grafana Loki
spec:
  type: loki_otlp
  parameters:
    - name: endpoint
      value: 'http://localhost:3100/otlp'
    - name: drop_raw_copy
      value: true
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true

Configuration with Basic Authentication

This example demonstrates a production-ready configuration with basic authentication.

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: loki-authenticated
  name: Grafana Loki (Authenticated)
spec:
  type: loki_otlp
  parameters:
    - name: endpoint
      value: 'https://loki.example.com/otlp'
    - name: use_basic_auth
      value: true
    - name: basic_auth_username
      value: 'loki-user'
    - name: basic_auth_password
      value: 'secure-password'
    - name: drop_raw_copy
      value: true
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: true

Secure Configuration with TLS and Basic Authentication

This example demonstrates a highly secure production configuration with TLS encryption and authentication.

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: loki-secure
  name: Grafana Loki (Secure)
spec:
  type: loki_otlp
  parameters:
    - name: endpoint
      value: 'https://loki.example.com/otlp'
    - name: use_basic_auth
      value: true
    - name: basic_auth_username
      value: 'loki-user'
    - name: basic_auth_password
      value: 'secure-password'
    - name: configure_tls
      value: true
    - name: insecure_skip_verify
      value: false
    - name: ca_file
      value: '/etc/ssl/certs/ca.crt'
    - name: drop_raw_copy
      value: true
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: true
    - name: persistent_queue_fsync
      value: true

Mutual TLS Configuration

This example shows how to configure mutual TLS (mTLS) with client certificates.

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: loki-mtls
  name: Grafana Loki (mTLS)
spec:
  type: loki_otlp
  parameters:
    - name: endpoint
      value: 'https://loki.example.com/otlp'
    - name: configure_tls
      value: true
    - name: insecure_skip_verify
      value: false
    - name: ca_file
      value: '/etc/ssl/certs/ca.crt'
    - name: mutual_tls
      value: true
    - name: cert_file
      value: '/etc/ssl/certs/client.crt'
    - name: key_file
      value: '/etc/ssl/private/client.key'
    - name: drop_raw_copy
      value: true
    - name: retry_on_failure_enabled
      value: true
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: true

Configuration Tips

Endpoint Configuration

• Use the format: http://hostname:port/otlp or https://hostname:port/otlp

• Default port for Loki OTLP is 3100

• The /otlp path is required in the endpoint URL

• Verify Loki is configured to accept OTLP logs

• Test the endpoint: curl -I http://localhost:3100/otlp

Raw Copy Handling

• Enable Drop Raw Copy to reduce storage by removing the log.record.original attribute

• Disable only if you need the original raw log record for debugging

• Default behavior (enabled) is recommended for production environments to save storage space

Security Best Practices

• Always use HTTPS in production environments

• Set Skip TLS Certificate Verification to false and provide a CA certificate when possible

• Use basic authentication only over HTTPS/TLS connections

• Consider mutual TLS for highly sensitive environments or zero-trust networks

• Store credentials securely using environment variables or secret management systems

• Regularly rotate credentials and certificates

• Use Grafana service accounts when available for better access control

Performance Tuning

• Increase Queue Size for high-volume log collection (default: 5000)

• Adjust Number of Consumers based on available CPU and network bandwidth

• Enable Persistent Queuing for mission-critical log collection

• Monitor queue depth and adjust settings if queues are consistently full

• Enable Synchronize Persistent Queue to Disk in critical environments (with performance trade-off)

• Disable Synchronize Persistent Queue to Disk for better performance when acceptable data loss risk is low

Network Considerations

• Ensure network connectivity between your Bindplane agent and the Grafana Loki endpoint

• Verify the OTLP HTTP receiver is enabled in your Loki configuration

• Open firewall rules for the appropriate port (default: 3100)

• Check if a proxy is required and configure accordingly

• Test connectivity before deployment: curl -I https://endpoint/otlp

---

Troubleshooting

Connection Refused

Symptoms: Logs are not being sent; connection timeout errors in logs.

Solutions:

1. Verify Grafana Loki is running: docker ps or check systemd status

2. Verify the endpoint URL is correct (should include /otlp path)

3. Test connectivity: curl -I http://endpoint:3100/otlp

4. Verify network firewall rules allow traffic on the port

5. Check Loki logs for errors

TLS Certificate Errors

Symptoms: "certificate verify failed" or "bad certificate" errors in logs.

Solutions:

1. Verify the CA certificate path exists and is readable

2. Check certificate validity: openssl x509 -in <cert> -noout -dates

3. Verify the certificate matches the hostname used in configuration

4. For self-signed certificates, ensure the CA certificate is installed correctly

5. Temporarily set Skip TLS Certificate Verification to true for testing (not recommended for production)

Authentication Failures

Symptoms: "401 Unauthorized" or authentication errors in logs.

Solutions:

1. Verify username and password are correct

2. Check that basic authentication is enabled on the Grafana Loki OTLP receiver

3. Ensure credentials are being transmitted over HTTPS/TLS

4. Verify credentials don't contain special characters that need escaping in YAML

Logs Not Appearing in Loki

Symptoms: Logs are sent but don't appear in Loki UI or queries.

Solutions:

1. Verify logs are being generated by the source

2. Check that Loki has at least one label configured (Loki requires labels for all logs)

3. Verify the OTLP receiver is configured correctly in Loki

4. Check Loki logs for ingestion errors or rejections

5. Verify log volume isn't exceeding Loki rate limits

High Queue Depth

Symptoms: Sending queue is consistently full; logs may be dropped if persistent queuing is disabled.

Solutions:

1. Verify Grafana Loki instance can handle the log volume

2. Check for network latency or bandwidth issues

3. Increase Queue Size to buffer more logs

4. Increase Number of Consumers to process batches faster

5. Enable Persistent Queuing to prevent log loss during outages

6. Consider distributing logs across multiple Loki instances

Performance Issues

Symptoms: High CPU or memory usage; slow log ingestion.

Solutions:

1. Monitor Bindplane agent CPU and memory usage

2. Consider reducing log volume at the source (sampling, filtering)

3. Disable Synchronize Persistent Queue to Disk if enabled (unless data integrity is critical)

4. Verify network connectivity and latency to Loki

5. Consider batching at the source to reduce log frequency

Data Loss

Symptoms: Logs are dropped after restarts or network outages.

Solutions:

1. Enable Persistent Queuing

2. Verify persistent queue directory exists and has write permissions

3. Monitor queue size to ensure it's not overflowing

4. Enable Retry on Failure to handle transient failures

5. Check available disk space for persistent queue storage

6. For critical environments, enable Synchronize Persistent Queue to Disk

7. Review logs for exporter errors or capacity issues

Related Resources

• Grafana Loki Documentation

• Grafana Loki OTLP Configuration

• Grafana Loki Setup and Installation

• OpenTelemetry Protocol Specification

• Bindplane Destinations Overview

• Retry and Queuing Configuration