# TLS Troubleshooting

## Quick Diagnosis

Use this table to quickly identify your issue and jump to the solution:

| Error Message / Symptom               | Likely Cause                                | Jump To                                                                                                 |
| ------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| "Certificate and key do not match"    | Certificate/key mismatch                    | [Certificate Mismatch](#issue-certificate-and-key-do-not-match-private-key-does-not-match-public-key)   |
| "Unknown CA" (from clients)           | Missing intermediate certificates           | [Unknown CA](#issue-clients-report-unknown-ca-or-certificate-verification-errors)                       |
| "Failed to load TLS cert and key"     | Encrypted key or wrong format               | [Failed to Load](#issue-collector-fails-to-start-with-failed-to-load-tls-cert-and-key-or-similar-error) |
| "TLS handshake timeout"               | TLS version/cipher incompatibility          | [Handshake Timeout](#issue-tls-handshake-timeout-or-connection-hangs)                                   |
| "Certificate required" (client error) | mTLS enforced, client has no cert           | [mTLS: Certificate Required](#mtls-issue-client-certificate-required-errors)                            |
| "Bad certificate" (client error)      | Client cert verification failed             | [mTLS: Verification Fails](#mtls-issue-client-certificate-verification-fails)                           |
| Clients connect without certs (mTLS)  | Using `ca_file` instead of `client_ca_file` | [mTLS: Connections Accepted](#mtls-issue-connections-accepted-without-client-certificates)              |
| Certificate expired errors            | Certificate past expiration date            | [Certificate Expired](#issue-certificate-expired)                                                       |

## Common Issues and Solutions

### Issue: Certificate and Key Do Not Match / Private Key Does Not Match Public Key

**Error Messages:**

* "Certificate and key do not match"
* "Private key does not match public key"
* "TLS handshake error"

**Cause:**

The certificate and private key files don't correspond to the same keypair. This commonly happens when:

* Using a certificate and key from different generation attempts
* Accidentally swapping certificate/key file paths in the configuration
* Using the wrong key file (e.g., `newcert2025.key` with `oldcert2024.crt`)
* Using an intermediate/root CA certificate instead of the server certificate

**Solution:**

**Step 1: Verify the files match**

Method 1 - Compare public keys (recommended):

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# Extract public key from private key
openssl rsa -in server.key -pubout -out key_public.pem

# Extract public key from certificate
openssl x509 -in server.crt -pubkey -noout -out cert_public.pem

# Compare (no output = they match)
diff key_public.pem cert_public.pem

# Or compare hashes
md5sum key_public.pem cert_public.pem
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# Extract public key from private key
openssl rsa -in C:\certs\server.key -pubout -out C:\certs\key_public.pem

# Extract public key from certificate
openssl x509 -in C:\certs\server.crt -pubkey -noout -out C:\certs\cert_public.pem

# Compare (no output = they match)
Compare-Object (Get-Content C:\certs\key_public.pem) (Get-Content C:\certs\cert_public.pem)

# Or compare hashes
Get-FileHash C:\certs\key_public.pem -Algorithm MD5
Get-FileHash C:\certs\cert_public.pem -Algorithm MD5
```

{% endtab %}
{% endtabs %}

Method 2 - Compare modulus hashes:

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# Get certificate modulus hash
openssl x509 -noout -modulus -in server.crt | openssl md5

# Get private key modulus hash
openssl rsa -noout -modulus -in server.key | openssl md5

# The MD5 hashes should be identical
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# Get certificate modulus hash
openssl x509 -noout -modulus -in C:\certs\server.crt | openssl md5

# Get private key modulus hash
openssl rsa -noout -modulus -in C:\certs\server.key | openssl md5

# The MD5 hashes should be identical
```

{% endtab %}
{% endtabs %}

**Step 2: If they don't match**

1. Check file naming - look for matching file names (e.g., if you have `bindplane-collector.crt`, look for `bindplane-collector.key`)
2. Verify you're using the leaf/server certificate, not an intermediate or root CA certificate
3. If no matching key exists, you'll need to either:
   * Locate the correct private key that was used to generate the CSR for this certificate
   * Generate a new private key and request a new certificate from your CA

For detailed verification steps, see [Testing and Verification: Verify Certificate and Key Match](/how-to-guides/security-and-tls/using-tls/reference/testing-verification.md#verifying-certificate-and-key-match).

***

### Issue: Clients Report "Unknown CA" or Certificate Verification Errors

**Error Messages (from clients):**

* "Unknown CA"
* "Certificate verification failed"
* "Unable to get issuer certificate"
* "SSL certificate problem: unable to get local issuer certificate"

**Cause:**

The collector is not sending the complete certificate chain. Clients need the full chain from your server certificate up to a trusted root CA.

**Solution:**

**Step 1: Obtain intermediate certificates**

Get the intermediate certificate(s) from your Certificate Authority (CA).

**Step 2: Create a fullchain certificate file**

The certificate file must contain certificates in this order:

1. Server/leaf certificate (first)
2. Intermediate CA certificate(s)
3. Root CA (optional, clients usually have this)

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# Concatenate certificates into a single file
cat server-cert.crt intermediate-ca.crt > fullchain.crt
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# Concatenate certificates into a single file
Get-Content C:\certs\server-cert.crt, C:\certs\intermediate-ca.crt | Set-Content C:\certs\fullchain.crt
```

{% endtab %}
{% endtabs %}

The resulting file should look like:

```
-----BEGIN CERTIFICATE-----
[Your Server Certificate]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[Intermediate CA Certificate]
-----END CERTIFICATE-----
```

**Step 3: Update collector configuration**

```yaml
tls:
  cert_file: /path/to/fullchain.crt  # Use the full chain
  key_file: /path/to/server.key
```

**Step 4: Restart the collector**

{% hint style="warning" %}
**Certificate Order Matters**

The order must be: leaf certificate first, then intermediate(s), optionally root CA last. Reverse order will cause verification failures.
{% endhint %}

For more details, see [Certificate Requirements: Certificate Chain](/how-to-guides/security-and-tls/using-tls/requirements.md#creating-a-full-chain-certificate).

***

### Issue: Collector Fails to Start with "Failed to Load TLS Cert and Key" or Similar Error

**Error Messages:**

* "Failed to load TLS cert and key"
* "TLS configuration error"
* "Could not load certificate"
* "Invalid key file"

**Common Causes:**

1. Private key file is encrypted (password-protected)
2. Certificate or key is in DER format instead of PEM
3. Certificate or key file is corrupted or malformed
4. Wrong file paths in configuration
5. File permission issues

**Solution:**

**Step 1: Verify PEM format**

Check if files are PEM format (should show BEGIN headers):

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
head -5 server.crt
head -5 server.key

# PEM certificates start with: -----BEGIN CERTIFICATE-----
# PEM keys start with: -----BEGIN PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY-----
# If you see binary data, your files are in DER format and must be converted
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
Get-Content C:\certs\server.crt -First 5
Get-Content C:\certs\server.key -First 5

# PEM certificates start with: -----BEGIN CERTIFICATE-----
# PEM keys start with: -----BEGIN PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY-----
# If you see binary data, your files are in DER format and must be converted
```

{% endtab %}
{% endtabs %}

**Step 2: Check if key is encrypted**

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
head -10 server.key

# Encrypted keys contain one of these:
# - Header: "-----BEGIN ENCRYPTED PRIVATE KEY-----"
# - Line: "Proc-Type: 4,ENCRYPTED"
# - Line: "DEK-Info:"
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
Get-Content C:\certs\server.key -First 10

# Encrypted keys contain one of these:
# - Header: "-----BEGIN ENCRYPTED PRIVATE KEY-----"
# - Line: "Proc-Type: 4,ENCRYPTED"
# - Line: "DEK-Info:"
```

{% endtab %}
{% endtabs %}

**Step 3: Decrypt if encrypted**

Bindplane collectors do NOT support encrypted keys:

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# For RSA keys (PKCS#1)
openssl rsa -in encrypted-server.key -out server.key
# You'll be prompted for the passphrase

# For other key types or PKCS#8 format
openssl pkey -in encrypted-server.key -out server.key

# Secure the decrypted key file
chmod 600 server.key
chown <collector-user>:<collector-group> server.key
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# For RSA keys (PKCS#1)
openssl rsa -in C:\certs\encrypted-server.key -out C:\certs\server.key
# You'll be prompted for the passphrase

# For other key types or PKCS#8 format
openssl pkey -in C:\certs\encrypted-server.key -out C:\certs\server.key

# Secure the decrypted key file
icacls C:\certs\server.key /inheritance:r /grant:r "$($env:USERNAME):(R)"
```

{% endtab %}
{% endtabs %}

**Step 4: Convert from DER to PEM if needed**

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# Convert DER certificate
openssl x509 -inform DER -in certificate.der -out certificate.pem

# Convert DER private key
openssl rsa -inform DER -in private.der -out private.pem
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# Convert DER certificate
openssl x509 -inform DER -in C:\certs\certificate.der -out C:\certs\certificate.pem

# Convert DER private key
openssl rsa -inform DER -in C:\certs\private.der -out C:\certs\private.pem
```

{% endtab %}
{% endtabs %}

**Step 5: Verify file permissions**

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# Private key should only be readable by the collector user
chmod 600 server.key
chown <collector-user>:<collector-group> server.key

# Certificate can be more permissive
chmod 644 server.crt
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# Private key should only be readable by the collector user
icacls C:\certs\server.key /inheritance:r /grant:r "$($env:USERNAME):(R)"

# Certificate can remain with default permissions
```

{% endtab %}
{% endtabs %}

For detailed format requirements, see [Certificate Requirements](/how-to-guides/security-and-tls/using-tls/requirements.md).

***

## Issue: TLS Handshake Timeout or Connection Hangs

**Symptoms:**

* Connection attempts hang or timeout
* "TLS handshake timeout" errors
* Connections drop during handshake

**Cause:**

Client and server TLS versions or cipher suites are incompatible.

**Solution:**

**Step 1: Verify minimum TLS version compatibility**

Check your collector configuration:

```yaml
tls:
  min_version: "1.2"  # Verify this is compatible with your clients
```

Older clients may only support TLS 1.0 or 1.1 (deprecated). Consider:

* Upgrading clients to support TLS 1.2 or higher (recommended)
* Temporarily lowering `min_version` for compatibility (not recommended for production)

**Step 2: Test with OpenSSL**

Test which TLS versions work:

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# Test TLS 1.2
openssl s_client -connect collector.example.com:10514 -tls1_2

# Test TLS 1.3
openssl s_client -connect collector.example.com:10514 -tls1_3
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# Test TLS 1.2
openssl s_client -connect collector.example.com:10514 -tls1_2

# Test TLS 1.3
openssl s_client -connect collector.example.com:10514 -tls1_3
```

{% endtab %}
{% endtabs %}

**Step 3: Review cipher suite compatibility**

If you've configured specific cipher suites, they may be incompatible with clients:

```yaml
tls:
  min_version: "1.2"
  # Remove or comment out cipher_suites to use defaults
  # cipher_suites:
  #   - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
```

{% hint style="success" %}
**Recommendation**

Use the default cipher suite list (don't specify `cipher_suites`) unless you have specific security requirements. The defaults are secure and broadly compatible.
{% endhint %}

***

### Issue: Certificate Expired

**Error Messages:**

* "Certificate has expired"
* "Certificate is not yet valid"
* "Certificate expired" (from clients)

**Cause:**

The server certificate has passed its expiration date or is not yet valid.

**Solution:**

**Step 1: Check certificate expiration**

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
openssl x509 -in server.crt -noout -dates
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
openssl x509 -in C:\certs\server.crt -noout -dates
```

{% endtab %}
{% endtabs %}

**Step 2: Obtain a renewed certificate**

Contact your CA or certificate provider to renew your certificate.

**Step 3: Update the certificate file**

Replace the old certificate with the new one, ensuring you maintain the full certificate chain if applicable.

**Step 4: Reload or restart the collector**

If using `reload_interval`, the certificate will be automatically reloaded. Otherwise, restart the collector:

```yaml
tls:
  cert_file: /path/to/fullchain.crt
  key_file: /path/to/server.key
  reload_interval: 24h  # Automatically check for updates every 24 hours
```

{% hint style="success" %}
**Best Practice**

Use `reload_interval` to automatically reload certificates without restarting the collector. This enables seamless certificate rotation.
{% endhint %}

***

### Issue: Trust Store Updates Don't Help Clients

**Misconception:** "I updated the collector's system trust store but clients still can't verify the certificate."

**Explanation:**

The collector's trust store only affects **outbound** connections the collector makes (e.g., to exporters). It does NOT affect:

* What certificate the collector presents to clients
* Whether clients can verify the collector's certificate

**Solution:**

To help clients verify the collector's certificate, you must update the `cert_file` with the proper certificate chain instead:

```yaml
tls:
  cert_file: /path/to/fullchain.crt  # Must include intermediates
  key_file: /path/to/server.key
```

See [Unknown CA](#issue-clients-report-unknown-ca-or-certificate-verification-errors) above for details.

***

## Mutual TLS (mTLS) Specific Issues

### mTLS Issue: Client Certificate Required Errors

**Error Messages (from clients):**

* "Certificate required"
* "Handshake failure"
* "TLS alert: certificate required"

**Cause:**

Client is not presenting a valid certificate when `client_ca_file` is configured on the collector.

**Solution:**

**Step 1: Verify client is presenting a certificate**

Test with OpenSSL:

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
openssl s_client -connect collector.example.com:10514 \
  -cert client.crt \
  -key client.key \
  -CAfile server-ca.crt
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
openssl s_client -connect collector.example.com:10514 `
  -cert C:\certs\client.crt `
  -key C:\certs\client.key `
  -CAfile C:\certs\server-ca.crt
```

{% endtab %}
{% endtabs %}

**Step 2: Ensure client certificate is signed by the trusted CA**

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
openssl verify -CAfile client-ca.crt client.crt
# Should show: client.crt: OK
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
openssl verify -CAfile C:\certs\client-ca.crt C:\certs\client.crt
# Should show: client.crt: OK
```

{% endtab %}
{% endtabs %}

**Step 3: Check certificate expiration**

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
openssl x509 -in client.crt -noout -enddate
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
openssl x509 -in C:\certs\client.crt -noout -enddate
```

{% endtab %}
{% endtabs %}

For more mTLS troubleshooting, see [Mutual TLS Guide: Troubleshooting](/how-to-guides/security-and-tls/using-tls/mutual-tls.md#mtls-troubleshooting).

***

### mTLS Issue: Client Certificate Verification Fails

**Error Messages (from clients):**

* "Bad certificate"
* "Unknown ca"
* "Certificate verification failed"

**Cause:**

The client CA file on the collector doesn't match the CA that signed the client certificate.

**Solution:**

**Step 1: Verify the CA file contains the correct CA**

Check that `client_ca_file` contains the CA that signed the client certificate:

```yaml
tls:
  client_ca_file: /path/to/client-ca.crt  # Must match CA that signed client certs
```

**Step 2: Verify the certificate chain**

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
openssl verify -CAfile client-ca.crt client.crt
# Should output: client.crt: OK
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
openssl verify -CAfile C:\certs\client-ca.crt C:\certs\client.crt
# Should output: client.crt: OK
```

{% endtab %}
{% endtabs %}

**Step 3: If using intermediate CAs, include the full chain**

The `client_ca_file` should contain the full CA chain if intermediates are involved:

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
cat intermediate-ca.crt root-ca.crt > client-ca-chain.crt
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
Get-Content C:\certs\intermediate-ca.crt, C:\certs\root-ca.crt | Set-Content C:\certs\client-ca-chain.crt
```

{% endtab %}
{% endtabs %}

***

### mTLS Issue: Connections Accepted Without Client Certificates

**Symptoms:**

* Clients without certificates can still connect
* mTLS is not being enforced
* No authentication is required

**Cause:**

Using `ca_file` instead of `client_ca_file`.

**Solution:**

**Verify your configuration uses `client_ca_file`:**

```yaml
# INCORRECT - does not enforce mTLS
tls:
  cert_file: /path/to/server.crt
  key_file: /path/to/server.key
  ca_file: /path/to/client-ca.crt  # Only optional verification

# CORRECT - enforces mTLS
tls:
  cert_file: /path/to/server.crt
  key_file: /path/to/server.key
  client_ca_file: /path/to/client-ca.crt  # Required verification
```

{% hint style="warning" %}
**Important Difference**

* `ca_file` provides **optional** client verification - clients can connect with or without certificates
* `client_ca_file` **enforces** mTLS - all clients must present valid certificates
  {% endhint %}

For more details, see [Mutual TLS Guide: Configuration Options](/how-to-guides/security-and-tls/using-tls/mutual-tls.md#mtls-configuration-options).

***

## Verification Commands Reference

### Check Certificate/Key Match

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# Method 1: Compare public keys (recommended)
openssl rsa -in server.key -pubout -out key_public.pem
openssl x509 -in server.crt -pubkey -noout -out cert_public.pem
diff key_public.pem cert_public.pem

# Method 2: Compare modulus hashes
openssl x509 -noout -modulus -in server.crt | openssl md5
openssl rsa -noout -modulus -in server.key | openssl md5
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# Method 1: Compare public keys (recommended)
openssl rsa -in C:\certs\server.key -pubout -out C:\certs\key_public.pem
openssl x509 -in C:\certs\server.crt -pubkey -noout -out C:\certs\cert_public.pem
Compare-Object (Get-Content C:\certs\key_public.pem) (Get-Content C:\certs\cert_public.pem)

# Method 2: Compare modulus hashes
openssl x509 -noout -modulus -in C:\certs\server.crt | openssl md5
openssl rsa -noout -modulus -in C:\certs\server.key | openssl md5
```

{% endtab %}
{% endtabs %}

### Verify Certificate Chain

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# View certificates in chain
openssl certs -in fullchain.crt -text -noout

# Test certificate chain with OpenSSL
openssl s_client -connect collector.example.com:10514 -showcerts
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# View certificates in chain
openssl certs -in C:\certs\fullchain.crt -text -noout

# Test certificate chain with OpenSSL
openssl s_client -connect collector.example.com:10514 -showcerts
```

{% endtab %}
{% endtabs %}

### Test TLS Connection

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# Basic connectivity test
openssl s_client -connect collector.example.com:10514

# Test with specific TLS version
openssl s_client -connect collector.example.com:10514 -tls1_2

# Test mTLS with client certificate
openssl s_client -connect collector.example.com:10514 \
  -cert client.crt \
  -key client.key \
  -CAfile server-ca.crt
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# Basic connectivity test
openssl s_client -connect collector.example.com:10514

# Test with specific TLS version
openssl s_client -connect collector.example.com:10514 -tls1_2

# Test mTLS with client certificate
openssl s_client -connect collector.example.com:10514 `
  -cert C:\certs\client.crt `
  -key C:\certs\client.key `
  -CAfile C:\certs\server-ca.crt
```

{% endtab %}
{% endtabs %}

### Inspect Certificate Details

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# View certificate information
openssl x509 -in server.crt -text -noout

# Check expiration date
openssl x509 -in server.crt -noout -enddate

# View subject and issuer
openssl x509 -in server.crt -noout -subject -issuer

# Verify client certificate against CA
openssl verify -CAfile client-ca.crt client.crt
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# View certificate information
openssl x509 -in C:\certs\server.crt -text -noout

# Check expiration date
openssl x509 -in C:\certs\server.crt -noout -enddate

# View subject and issuer
openssl x509 -in C:\certs\server.crt -noout -subject -issuer

# Verify client certificate against CA
openssl verify -CAfile C:\certs\client-ca.crt C:\certs\client.crt
```

{% endtab %}
{% endtabs %}

For more verification commands, see [Testing and Verification](/how-to-guides/security-and-tls/using-tls/reference/testing-verification.md).

## When to Seek Further Help

If you've tried the solutions above and still have issues, gather this information before contacting support:

**Collector Information:**

* Bindplane collector version
* Operating system and version
* Collector configuration (TLS section, with sensitive paths redacted)

**Certificate Information:**

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# Certificate details
openssl x509 -in server.crt -text -noout

# Check if key is encrypted
head -10 server.key

# Verify cert/key match
openssl x509 -in server.crt -noout -modulus | openssl md5
openssl rsa -in server.key -noout -modulus | openssl md5
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# Certificate details
openssl x509 -in C:\certs\server.crt -text -noout

# Check if key is encrypted
Get-Content C:\certs\server.key -First 10

# Verify cert/key match
openssl x509 -in C:\certs\server.crt -noout -modulus | openssl md5
openssl rsa -in C:\certs\server.key -noout -modulus | openssl md5
```

{% endtab %}
{% endtabs %}

**Error Information:**

* Exact error messages from collector logs
* Exact error messages from clients
* When the issue started occurring
* Any recent changes to certificates or configuration

**Test Results:**

{% tabs %}
{% tab title="Linux/macOS" %}

```bash
# OpenSSL connection test output
openssl s_client -connect collector.example.com:10514
```

{% endtab %}

{% tab title="Windows PowerShell" %}

```powershell
# OpenSSL connection test output
openssl s_client -connect collector.example.com:10514
```

{% endtab %}
{% endtabs %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.bindplane.com/how-to-guides/security-and-tls/using-tls/troubleshooting.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.