TLS Troubleshooting

Solutions to common TLS issues when configuring Bindplane collector receivers.

Quick Diagnosis

Use this table to quickly identify your issue and jump to the solution:

Error Message / Symptom	Likely Cause	Jump To
"Certificate and key do not match"	Certificate/key mismatch	Certificate Mismatch
"Unknown CA" (from clients)	Missing intermediate certificates	Unknown CA
"Failed to load TLS cert and key"	Encrypted key or wrong format	Failed to Load
"TLS handshake timeout"	TLS version/cipher incompatibility	Handshake Timeout
"Certificate required" (client error)	mTLS enforced, client has no cert	mTLS: Certificate Required
"Bad certificate" (client error)	Client cert verification failed	mTLS: Verification Fails
Clients connect without certs (mTLS)	Using ca_file instead of client_ca_file	mTLS: Connections Accepted
Certificate expired errors	Certificate past expiration date	Certificate Expired

Common Issues and Solutions

Issue: Certificate and Key Do Not Match / Private Key Does Not Match Public Key

Error Messages:

• "Certificate and key do not match"

• "Private key does not match public key"

• "TLS handshake error"

Cause:

The certificate and private key files don't correspond to the same keypair. This commonly happens when:

• Using a certificate and key from different generation attempts

• Accidentally swapping certificate/key file paths in the configuration

• Using the wrong key file (e.g., newcert2025.key with oldcert2024.crt)

• Using an intermediate/root CA certificate instead of the server certificate

Solution:

Step 1: Verify the files match

Method 1 - Compare public keys (recommended):

Linux/macOS

# Extract public key from private key
openssl rsa -in server.key -pubout -out key_public.pem

# Extract public key from certificate
openssl x509 -in server.crt -pubkey -noout -out cert_public.pem

# Compare (no output = they match)
diff key_public.pem cert_public.pem

# Or compare hashes
md5sum key_public.pem cert_public.pem

Windows PowerShell

Method 2 - Compare modulus hashes:

Linux/macOS

# Get certificate modulus hash
openssl x509 -noout -modulus -in server.crt | openssl md5

# Get private key modulus hash
openssl rsa -noout -modulus -in server.key | openssl md5

# The MD5 hashes should be identical

Windows PowerShell

Step 2: If they don't match

1. Check file naming - look for matching file names (e.g., if you have bindplane-collector.crt, look for bindplane-collector.key)

2. Verify you're using the leaf/server certificate, not an intermediate or root CA certificate

3. If no matching key exists, you'll need to either:

   • Locate the correct private key that was used to generate the CSR for this certificate

   • Generate a new private key and request a new certificate from your CA

For detailed verification steps, see Testing and Verification: Verify Certificate and Key Match.

---

Issue: Clients Report "Unknown CA" or Certificate Verification Errors

Error Messages (from clients):

• "Unknown CA"

• "Certificate verification failed"

• "Unable to get issuer certificate"

• "SSL certificate problem: unable to get local issuer certificate"

Cause:

The collector is not sending the complete certificate chain. Clients need the full chain from your server certificate up to a trusted root CA.

Solution:

Step 1: Obtain intermediate certificates

Get the intermediate certificate(s) from your Certificate Authority (CA).

Step 2: Create a fullchain certificate file

The certificate file must contain certificates in this order:

1. Server/leaf certificate (first)

2. Intermediate CA certificate(s)

3. Root CA (optional, clients usually have this)

Linux/macOS

# Concatenate certificates into a single file
cat server-cert.crt intermediate-ca.crt > fullchain.crt

Windows PowerShell

The resulting file should look like:

-----BEGIN CERTIFICATE-----
[Your Server Certificate]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[Intermediate CA Certificate]
-----END CERTIFICATE-----

Step 3: Update collector configuration

tls:
  cert_file: /path/to/fullchain.crt  # Use the full chain
  key_file: /path/to/server.key

Step 4: Restart the collector

Certificate Order Matters

The order must be: leaf certificate first, then intermediate(s), optionally root CA last. Reverse order will cause verification failures.

For more details, see Certificate Requirements: Certificate Chain.

---

Issue: Collector Fails to Start with "Failed to Load TLS Cert and Key" or Similar Error

Error Messages:

• "Failed to load TLS cert and key"

• "TLS configuration error"

• "Could not load certificate"

• "Invalid key file"

Common Causes:

1. Private key file is encrypted (password-protected)

2. Certificate or key is in DER format instead of PEM

3. Certificate or key file is corrupted or malformed

4. Wrong file paths in configuration

5. File permission issues

Solution:

Step 1: Verify PEM format

Check if files are PEM format (should show BEGIN headers):

Linux/macOS

head -5 server.crt
head -5 server.key

# PEM certificates start with: -----BEGIN CERTIFICATE-----
# PEM keys start with: -----BEGIN PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY-----
# If you see binary data, your files are in DER format and must be converted

Windows PowerShell

Step 2: Check if key is encrypted

Linux/macOS

head -10 server.key

# Encrypted keys contain one of these:
# - Header: "-----BEGIN ENCRYPTED PRIVATE KEY-----"
# - Line: "Proc-Type: 4,ENCRYPTED"
# - Line: "DEK-Info:"

Windows PowerShell

Step 3: Decrypt if encrypted

Bindplane collectors do NOT support encrypted keys:

Linux/macOS

# For RSA keys (PKCS#1)
openssl rsa -in encrypted-server.key -out server.key
# You'll be prompted for the passphrase

# For other key types or PKCS#8 format
openssl pkey -in encrypted-server.key -out server.key

# Secure the decrypted key file
chmod 600 server.key
chown <collector-user>:<collector-group> server.key

Windows PowerShell

Step 4: Convert from DER to PEM if needed

Linux/macOS

# Convert DER certificate
openssl x509 -inform DER -in certificate.der -out certificate.pem

# Convert DER private key
openssl rsa -inform DER -in private.der -out private.pem

Windows PowerShell

Step 5: Verify file permissions

Linux/macOS

# Private key should only be readable by the collector user
chmod 600 server.key
chown <collector-user>:<collector-group> server.key

# Certificate can be more permissive
chmod 644 server.crt

Windows PowerShell

For detailed format requirements, see Certificate Requirements.

---

Issue: TLS Handshake Timeout or Connection Hangs

Symptoms:

• Connection attempts hang or timeout

• "TLS handshake timeout" errors

• Connections drop during handshake

Cause:

Client and server TLS versions or cipher suites are incompatible.

Solution:

Step 1: Verify minimum TLS version compatibility

Check your collector configuration:

tls:
  min_version: "1.2"  # Verify this is compatible with your clients

Older clients may only support TLS 1.0 or 1.1 (deprecated). Consider:

• Upgrading clients to support TLS 1.2 or higher (recommended)

• Temporarily lowering min_version for compatibility (not recommended for production)

Step 2: Test with OpenSSL

Test which TLS versions work:

Linux/macOS

# Test TLS 1.2
openssl s_client -connect collector.example.com:10514 -tls1_2

# Test TLS 1.3
openssl s_client -connect collector.example.com:10514 -tls1_3

Windows PowerShell

Step 3: Review cipher suite compatibility

If you've configured specific cipher suites, they may be incompatible with clients:

tls:
  min_version: "1.2"
  # Remove or comment out cipher_suites to use defaults
  # cipher_suites:
  #   - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Recommendation

Use the default cipher suite list (don't specify cipher_suites) unless you have specific security requirements. The defaults are secure and broadly compatible.

---

Issue: Certificate Expired

Error Messages:

• "Certificate has expired"

• "Certificate is not yet valid"

• "Certificate expired" (from clients)

Cause:

The server certificate has passed its expiration date or is not yet valid.

Solution:

Step 1: Check certificate expiration

Linux/macOS

openssl x509 -in server.crt -noout -dates

Windows PowerShell

Step 2: Obtain a renewed certificate

Contact your CA or certificate provider to renew your certificate.

Step 3: Update the certificate file

Replace the old certificate with the new one, ensuring you maintain the full certificate chain if applicable.

Step 4: Reload or restart the collector

If using reload_interval, the certificate will be automatically reloaded. Otherwise, restart the collector:

tls:
  cert_file: /path/to/fullchain.crt
  key_file: /path/to/server.key
  reload_interval: 24h  # Automatically check for updates every 24 hours

Best Practice

Use reload_interval to automatically reload certificates without restarting the collector. This enables seamless certificate rotation.

---

Issue: Trust Store Updates Don't Help Clients

Misconception: "I updated the collector's system trust store but clients still can't verify the certificate."

Explanation:

The collector's trust store only affects outbound connections the collector makes (e.g., to exporters). It does NOT affect:

• What certificate the collector presents to clients

• Whether clients can verify the collector's certificate

Solution:

To help clients verify the collector's certificate, you must update the cert_file with the proper certificate chain instead:

tls:
  cert_file: /path/to/fullchain.crt  # Must include intermediates
  key_file: /path/to/server.key

See Unknown CA above for details.

---

Mutual TLS (mTLS) Specific Issues

mTLS Issue: Client Certificate Required Errors

Error Messages (from clients):

• "Certificate required"

• "Handshake failure"

• "TLS alert: certificate required"

Cause:

Client is not presenting a valid certificate when client_ca_file is configured on the collector.

Solution:

Step 1: Verify client is presenting a certificate

Test with OpenSSL:

Linux/macOS

openssl s_client -connect collector.example.com:10514 \
  -cert client.crt \
  -key client.key \
  -CAfile server-ca.crt

Windows PowerShell

Step 2: Ensure client certificate is signed by the trusted CA

Linux/macOS

openssl verify -CAfile client-ca.crt client.crt
# Should show: client.crt: OK

Windows PowerShell

Step 3: Check certificate expiration

Linux/macOS

openssl x509 -in client.crt -noout -enddate

Windows PowerShell

For more mTLS troubleshooting, see Mutual TLS Guide: Troubleshooting.

---

mTLS Issue: Client Certificate Verification Fails

Error Messages (from clients):

• "Bad certificate"

• "Unknown ca"

• "Certificate verification failed"

Cause:

The client CA file on the collector doesn't match the CA that signed the client certificate.

Solution:

Step 1: Verify the CA file contains the correct CA

Check that client_ca_file contains the CA that signed the client certificate:

tls:
  client_ca_file: /path/to/client-ca.crt  # Must match CA that signed client certs

Step 2: Verify the certificate chain

Linux/macOS

openssl verify -CAfile client-ca.crt client.crt
# Should output: client.crt: OK

Windows PowerShell

Step 3: If using intermediate CAs, include the full chain

The client_ca_file should contain the full CA chain if intermediates are involved:

Linux/macOS

cat intermediate-ca.crt root-ca.crt > client-ca-chain.crt

Windows PowerShell

---

mTLS Issue: Connections Accepted Without Client Certificates

Symptoms:

• Clients without certificates can still connect

• mTLS is not being enforced

• No authentication is required

Cause:

Using ca_file instead of client_ca_file.

Solution:

Verify your configuration uses client_ca_file:

# INCORRECT - does not enforce mTLS
tls:
  cert_file: /path/to/server.crt
  key_file: /path/to/server.key
  ca_file: /path/to/client-ca.crt  # Only optional verification

# CORRECT - enforces mTLS
tls:
  cert_file: /path/to/server.crt
  key_file: /path/to/server.key
  client_ca_file: /path/to/client-ca.crt  # Required verification

Important Difference

• ca_file provides optional client verification - clients can connect with or without certificates

• client_ca_file enforces mTLS - all clients must present valid certificates

For more details, see Mutual TLS Guide: Configuration Options.

---

Verification Commands Reference

Check Certificate/Key Match

Linux/macOS

# Method 1: Compare public keys (recommended)
openssl rsa -in server.key -pubout -out key_public.pem
openssl x509 -in server.crt -pubkey -noout -out cert_public.pem
diff key_public.pem cert_public.pem

# Method 2: Compare modulus hashes
openssl x509 -noout -modulus -in server.crt | openssl md5
openssl rsa -noout -modulus -in server.key | openssl md5

Windows PowerShell

Verify Certificate Chain

Linux/macOS

# View certificates in chain
openssl certs -in fullchain.crt -text -noout

# Test certificate chain with OpenSSL
openssl s_client -connect collector.example.com:10514 -showcerts

Windows PowerShell

Test TLS Connection

Linux/macOS

# Basic connectivity test
openssl s_client -connect collector.example.com:10514

# Test with specific TLS version
openssl s_client -connect collector.example.com:10514 -tls1_2

# Test mTLS with client certificate
openssl s_client -connect collector.example.com:10514 \
  -cert client.crt \
  -key client.key \
  -CAfile server-ca.crt

Windows PowerShell

Inspect Certificate Details

Linux/macOS

# View certificate information
openssl x509 -in server.crt -text -noout

# Check expiration date
openssl x509 -in server.crt -noout -enddate

# View subject and issuer
openssl x509 -in server.crt -noout -subject -issuer

# Verify client certificate against CA
openssl verify -CAfile client-ca.crt client.crt

Windows PowerShell

For more verification commands, see Testing and Verification.

When to Seek Further Help

If you've tried the solutions above and still have issues, gather this information before contacting support:

Collector Information:

• Bindplane collector version

• Operating system and version

• Collector configuration (TLS section, with sensitive paths redacted)

Certificate Information:

Linux/macOS

# Certificate details
openssl x509 -in server.crt -text -noout

# Check if key is encrypted
head -10 server.key

# Verify cert/key match
openssl x509 -in server.crt -noout -modulus | openssl md5
openssl rsa -in server.key -noout -modulus | openssl md5

Windows PowerShell

Error Information:

• Exact error messages from collector logs

• Exact error messages from clients

• When the issue started occurring

• Any recent changes to certificates or configuration

Test Results:

Linux/macOS

# OpenSSL connection test output
openssl s_client -connect collector.example.com:10514

Windows PowerShell