# Testing and Verification ## Testing TLS Connections with OpenSSL OpenSSL's `s_client` tool is the standard for testing TLS connections. It allows you to inspect certificates, test handshakes, and debug TLS issues. ### Basic Connectivity Test Test that the TLS server is accepting connections: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl s_client -connect collector.example.com:10514 ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl s_client -connect collector.example.com:10514 ``` {% endtab %} {% endtabs %} **What to look for:** * `CONNECTED` - TCP connection established * Certificate chain displayed * `Verify return code: 0 (ok)` - Certificate verified successfully * Or appropriate verification result based on your setup **To exit:** Type `CTRL-C` or `Q` then `ENTER` ### View Certificate Chain See the complete certificate chain sent by the server: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl s_client -connect collector.example.com:10514 -showcerts ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl s_client -connect collector.example.com:10514 -showcerts ``` {% endtab %} {% endtabs %} **What this shows:** * All certificates in the chain (server cert + intermediates) * Full PEM-encoded certificates * Certificate details for each cert in the chain **Useful for:** * Verifying the server sends intermediate certificates * Checking certificate order * Debugging "Unknown CA" errors ### Test Specific TLS Version Test connectivity with a specific TLS version: {% tabs %} {% tab title="Linux/macOS" %} ```bash # Test TLS 1.2 openssl s_client -connect collector.example.com:10514 -tls1_2 # Test TLS 1.3 openssl s_client -connect collector.example.com:10514 -tls1_3 ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # Test TLS 1.2 openssl s_client -connect collector.example.com:10514 -tls1_2 # Test TLS 1.3 openssl s_client -connect collector.example.com:10514 -tls1_3 ``` {% endtab %} {% endtabs %} **Expected results:** * Connection succeeds if version is supported * Connection fails if version is not supported (below `min_version` or above `max_version`) ### Test Cipher Suites Test a specific cipher suite: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl s_client -connect collector.example.com:10514 \ -cipher 'ECDHE-RSA-AES256-GCM-SHA384' ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl s_client -connect collector.example.com:10514 ` -cipher 'ECDHE-RSA-AES256-GCM-SHA384' ``` {% endtab %} {% endtabs %} **List supported cipher suites:** {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl s_client -connect collector.example.com:10514 -showcerts \ 2>&1 | grep "Cipher" ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl s_client -connect collector.example.com:10514 -showcerts 2>&1 | Select-String "Cipher" ``` {% endtab %} {% endtabs %} ### OpenSSL s\_client Options Reference | Option | Description | | ------------------------ | ----------------------------------------- | | `-connect :` | Server to connect to | | `-showcerts` | Show all certificates in the chain | | `-CAfile ` | CA certificate file for verification | | `-cert ` | Client certificate file (for mTLS) | | `-key ` | Client private key file (for mTLS) | | `-tls1_2` | Use only TLS 1.2 | | `-tls1_3` | Use only TLS 1.3 | | `-cipher ` | Specify cipher suites to use | | `-servername ` | Set SNI (Server Name Indication) hostname | | `-debug` | Print debug information | | `-state` | Print TLS session states | For complete OpenSSL s\_client documentation, see: ## Verifying Certificate and Key Match It's critical to verify that your certificate and private key are a matching pair before deploying them. ### Method 1: Compare Public Keys (Recommended) This method extracts the public key from both files and compares them directly: {% tabs %} {% tab title="Linux/macOS" %} ```bash # Extract public key from the private key openssl rsa -in server.key -pubout -out key_public.pem # Extract public key from the certificate openssl x509 -in server.crt -pubkey -noout -out cert_public.pem # Compare the two public keys (no output = they match) diff key_public.pem cert_public.pem # Or compare file hashes (Linux/macOS) md5sum key_public.pem cert_public.pem # Hashes should be identical # macOS alternative md5 key_public.pem cert_public.pem ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # Extract public keys openssl rsa -in server.key -pubout -out key_public.pem openssl x509 -in server.crt -pubkey -noout -out cert_public.pem # Compare file hashes (Get-FileHash key_public.pem).Hash -eq (Get-FileHash cert_public.pem).Hash # Should return: True ``` {% endtab %} {% endtabs %} **Interpretation:** * No output from `diff` = Files match * Identical hashes = Files match * Any difference = Files do NOT match ### Method 2: Compare Modulus Hashes This method compares the modulus values from both files: {% tabs %} {% tab title="Linux/macOS" %} ```bash # Get certificate modulus hash openssl x509 -noout -modulus -in server.crt | openssl md5 # Get private key modulus hash openssl rsa -noout -modulus -in server.key | openssl md5 # The MD5 hashes should be identical ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl x509 -in server.crt -noout -modulus | openssl md5 openssl rsa -in server.key -noout -modulus | openssl md5 ``` {% endtab %} {% endtabs %} **Example output:** ``` (stdin)= 8d4e2a5f9c8b3e1a7d6c4f2e8b9a5c3d # Certificate modulus (stdin)= 8d4e2a5f9c8b3e1a7d6c4f2e8b9a5c3d # Key modulus (should match) ``` {% hint style="success" %} **Best Practice** Use Method 1 (Compare Public Keys) as it's more reliable and works with all key types (RSA, ECDSA, Ed25519). {% endhint %} ## Certificate Inspection ### View Certificate Details Inspect all information in a certificate: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl x509 -in server.crt -text -noout ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl x509 -in C:\certs\server.crt -text -noout ``` {% endtab %} {% endtabs %} **Key information shown:** * Subject (who the certificate identifies) * Issuer (who signed the certificate) * Validity period (not before / not after dates) * Public key algorithm and size * Subject Alternative Names (SANs) * Key usage and extended key usage * Signature algorithm ### Check Expiration Date Quickly check when a certificate expires: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl x509 -in server.crt -noout -enddate ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl x509 -in C:\certs\server.crt -noout -enddate ``` {% endtab %} {% endtabs %} **Example output:** ``` notAfter=Jan 15 23:59:59 2025 GMT ``` **Check both start and end dates:** {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl x509 -in server.crt -noout -dates ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl x509 -in C:\certs\server.crt -noout -dates ``` {% endtab %} {% endtabs %} ### View Subject and Issuer See who the certificate identifies and who issued it: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl x509 -in server.crt -noout -subject -issuer ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl x509 -in C:\certs\server.crt -noout -subject -issuer ``` {% endtab %} {% endtabs %} **Example output:** ``` subject=CN = collector.example.com, O = Example Org issuer=CN = Example Intermediate CA, O = Example Org ``` ### View Subject Alternative Names (SANs) See all hostnames/IPs the certificate is valid for: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl x509 -in server.crt -noout -ext subjectAltName ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl x509 -in C:\certs\server.crt -noout -ext subjectAltName ``` {% endtab %} {% endtabs %} **Example output:** ``` X509v3 Subject Alternative Name: DNS:collector.example.com, DNS:*.example.com, IP Address:192.168.1.10 ``` ### Check Certificate Chain Order When you have a chain file, verify the order: {% tabs %} {% tab title="Linux/macOS" %} ```bash # View all certificates in the chain openssl certs -in fullchain.crt -text -noout # Or use a more detailed approach awk 'BEGIN {c=0} /BEGIN CERT/{c++} {print > "cert" c ".pem"}' fullchain.crt openssl x509 -in cert1.pem -noout -subject -issuer openssl x509 -in cert2.pem -noout -subject -issuer ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # View all certificates in the chain openssl certs -in C:\certs\fullchain.crt -text -noout # Or use a more detailed approach - split chain into individual certs $content = Get-Content C:\certs\fullchain.crt -Raw $certs = [regex]::Matches($content, '(?s)(-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----)') for ($i = 0; $i -lt $certs.Count; $i++) { $certs[$i].Value | Set-Content "cert$($i+1).pem" } openssl x509 -in cert1.pem -noout -subject -issuer openssl x509 -in cert2.pem -noout -subject -issuer ``` {% endtab %} {% endtabs %} **Expected order:** 1. First certificate: Server/leaf (subject = your server) 2. Second certificate: Intermediate CA (subject = intermediate, issuer = root or another intermediate) 3. Third certificate (optional): Root CA (subject = issuer = root CA) ## Testing Mutual TLS (mTLS) ### Test with Client Certificate Test mTLS connection with a client certificate: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl s_client -connect collector.example.com:10514 \ -cert client.crt \ -key client.key \ -CAfile server-ca.crt ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl s_client -connect collector.example.com:10514 ` -cert client.crt ` -key client.key ` -CAfile server-ca.crt ``` {% endtab %} {% endtabs %} **What to look for:** * `CONNECTED` - Connection established * `Verify return code: 0 (ok)` - Server certificate verified * Client certificate sent (shown in output) * Connection successful ### Verify Unauthorized Clients Are Rejected When using `client_ca_file`, test that connections without client certificates fail: {% tabs %} {% tab title="Linux/macOS" %} ```bash # This should fail with "certificate required" error openssl s_client -connect collector.example.com:10514 ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # This should fail with "certificate required" error openssl s_client -connect collector.example.com:10514 ``` {% endtab %} {% endtabs %} **Expected result:** * Connection rejected during handshake * Error: `certificate required` or `handshake failure` * Exit code: non-zero ### Debug Client Certificate Issues Get detailed information about client certificate verification: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl s_client -connect collector.example.com:10514 \ -cert client.crt \ -key client.key \ -CAfile server-ca.crt \ -state \ -debug ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl s_client -connect collector.example.com:10514 ` -cert client.crt ` -key client.key ` -CAfile server-ca.crt ` -state ` -debug ``` {% endtab %} {% endtabs %} **Options:** * `-state`: Shows TLS protocol state transitions * `-debug`: Shows detailed protocol messages ### Verify Client Certificate Against CA Check that a client certificate is properly signed by the CA: {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl verify -CAfile client-ca.crt client.crt ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl verify -CAfile client-ca.crt client.crt ``` {% endtab %} {% endtabs %} **Expected output:** ``` client.crt: OK ``` **If verification fails:** * `unable to get issuer certificate` - CA file doesn't contain the issuer * `certificate has expired` - Client certificate is expired * `unable to get local issuer certificate` - Missing intermediate CA ## Platform-Specific Commands ### Check Certificate Format {% tabs %} {% tab title="Linux/macOS" %} ```bash file certificate.crt # PEM: "PEM certificate" # DER: "DER encoded certificate" ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell Get-Content C:\certs\certificate.crt -First 1 # If output starts with "-----BEGIN" it is PEM format # If output is binary/unreadable it is DER format ``` {% endtab %} {% endtabs %} ### View First Few Lines {% tabs %} {% tab title="Linux/macOS" %} ```bash head -n 5 certificate.crt ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell Get-Content C:\certs\certificate.crt -First 5 ``` {% endtab %} {% endtabs %} ### Calculate File Hashes {% tabs %} {% tab title="Linux/macOS" %} ```bash md5sum certificate.crt sha256sum certificate.crt ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell Get-FileHash C:\certs\certificate.crt -Algorithm MD5 Get-FileHash C:\certs\certificate.crt -Algorithm SHA256 ``` {% endtab %} {% endtabs %} ### Test OpenSSL Installation {% tabs %} {% tab title="Linux/macOS" %} ```bash openssl version ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell openssl version ``` {% endtab %} {% endtabs %} ## Automated Testing ### Health Check Script Create a simple health check script to verify TLS connectivity: {% tabs %} {% tab title="Linux/macOS" %} ```bash #!/bin/bash # tls-health-check.sh HOST="collector.example.com" PORT="10514" TIMEOUT=5 echo "Testing TLS connection to $HOST:$PORT..." # Test connection with timeout timeout $TIMEOUT openssl s_client -connect $HOST:$PORT &1 | \ grep -q "Verify return code: 0 (ok)" if [ $? -eq 0 ]; then echo "TLS connection successful" exit 0 else echo "TLS connection failed" exit 1 fi ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # tls-health-check.ps1 $Host_ = "collector.example.com" $Port = "10514" Write-Host "Testing TLS connection to ${Host_}:${Port}..." # Test connection $result = echo "" | openssl s_client -connect "${Host_}:${Port}" 2>&1 if ($result -match "Verify return code: 0 \(ok\)") { Write-Host "TLS connection successful" exit 0 } else { Write-Host "TLS connection failed" exit 1 } ``` {% endtab %} {% endtabs %} **Usage:** {% tabs %} {% tab title="Linux/macOS" %} ```bash chmod +x tls-health-check.sh ./tls-health-check.sh ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell .\tls-health-check.ps1 ``` {% endtab %} {% endtabs %} ### Certificate Expiration Monitoring Monitor certificate expiration dates: {% tabs %} {% tab title="Linux/macOS" %} ```bash #!/bin/bash # cert-expiry-check.sh CERT_FILE="server.crt" WARNING_DAYS=30 # Get expiration date EXPIRY_DATE=$(openssl x509 -in $CERT_FILE -noout -enddate | cut -d= -f2) EXPIRY_EPOCH=$(date -d "$EXPIRY_DATE" +%s 2>/dev/null || date -j -f "%b %d %T %Y %Z" "$EXPIRY_DATE" +%s) NOW_EPOCH=$(date +%s) DAYS_LEFT=$(( ($EXPIRY_EPOCH - $NOW_EPOCH) / 86400 )) echo "Certificate expires: $EXPIRY_DATE" echo "Days remaining: $DAYS_LEFT" if [ $DAYS_LEFT -lt 0 ]; then echo "Certificate has EXPIRED" exit 2 elif [ $DAYS_LEFT -lt $WARNING_DAYS ]; then echo "Certificate expires in less than $WARNING_DAYS days" exit 1 else echo "Certificate is valid" exit 0 fi ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # cert-expiry-check.ps1 $CertFile = "C:\certs\server.crt" $WarningDays = 30 # Get expiration date $expiryLine = openssl x509 -in $CertFile -noout -enddate $expiryDate = [datetime]::Parse(($expiryLine -split "=")[1]) $daysLeft = ($expiryDate - (Get-Date)).Days Write-Host "Certificate expires: $expiryDate" Write-Host "Days remaining: $daysLeft" if ($daysLeft -lt 0) { Write-Host "Certificate has EXPIRED" exit 2 } elseif ($daysLeft -lt $WarningDays) { Write-Host "Certificate expires in less than $WarningDays days" exit 1 } else { Write-Host "Certificate is valid" exit 0 } ``` {% endtab %} {% endtabs %} ### Integration Testing Test the full pipeline with sample data: {% tabs %} {% tab title="Linux/macOS" %} ```bash #!/bin/bash # tls-integration-test.sh HOST="collector.example.com" PORT="10514" MESSAGE="Test log message from integration test" # Send test message over TLS echo "$MESSAGE" | openssl s_client -connect $HOST:$PORT -quiet 2>/dev/null if [ $? -eq 0 ]; then echo "Test message sent successfully" else echo "Failed to send test message" exit 1 fi ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # tls-integration-test.ps1 $Host_ = "collector.example.com" $Port = "10514" $Message = "Test log message from integration test" # Send test message over TLS $Message | openssl s_client -connect "${Host_}:${Port}" -quiet 2>$null if ($LASTEXITCODE -eq 0) { Write-Host "Test message sent successfully" } else { Write-Host "Failed to send test message" exit 1 } ``` {% endtab %} {% endtabs %} ## Continuous Monitoring ### Certificate Renewal Checks Set up periodic checks for certificate renewal: {% tabs %} {% tab title="Linux/macOS" %} ```bash # Add to crontab (runs daily at 2 AM) 0 2 * * * /path/to/cert-expiry-check.sh >> /var/log/cert-expiry.log 2>&1 ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # Create a scheduled task (runs daily at 2 AM) $action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-File C:\scripts\cert-expiry-check.ps1" $trigger = New-ScheduledTaskTrigger -Daily -At 2am Register-ScheduledTask -TaskName "CertExpiryCheck" -Action $action -Trigger $trigger ``` {% endtab %} {% endtabs %} ### TLS Endpoint Monitoring Use tools like: * **Nagios/Icinga**: Check certificate expiration and validity * **Prometheus**: blackbox\_exporter for TLS monitoring * **Datadog/New Relic**: Synthetic monitoring for TLS endpoints **Example Prometheus blackbox\_exporter config:** ```yaml modules: tls_connect: prober: tcp timeout: 5s tcp: tls: true tls_config: insecure_skip_verify: false ``` ## Common Verification Scenarios ### Verify Certificate Chain is Complete {% tabs %} {% tab title="Linux/macOS" %} ```bash # Test from a client's perspective openssl s_client -connect collector.example.com:10514 -showcerts # Look for the full chain in output # Should see: Server cert → Intermediate(s) → (optionally) Root ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # Test from a client's perspective openssl s_client -connect collector.example.com:10514 -showcerts # Look for the full chain in output # Should see: Server cert -> Intermediate(s) -> (optionally) Root ``` {% endtab %} {% endtabs %} ### Verify Certificate Matches Domain {% tabs %} {% tab title="Linux/macOS" %} ```bash # Check Subject Alternative Names (SANs) openssl x509 -in server.crt -noout -ext subjectAltName # Or check Common Name (CN) openssl x509 -in server.crt -noout -subject ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # Check Subject Alternative Names (SANs) openssl x509 -in C:\certs\server.crt -noout -ext subjectAltName # Or check Common Name (CN) openssl x509 -in C:\certs\server.crt -noout -subject ``` {% endtab %} {% endtabs %} ### Verify Private Key is Unencrypted {% tabs %} {% tab title="Linux/macOS" %} ```bash # Check key file headers head -n 10 server.key # Should see: # -----BEGIN PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY----- # Should NOT see: # -----BEGIN ENCRYPTED PRIVATE KEY----- or "Proc-Type: 4,ENCRYPTED" ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # Check key file headers Get-Content C:\certs\server.key -First 10 # Should see: # -----BEGIN PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY----- # Should NOT see: # -----BEGIN ENCRYPTED PRIVATE KEY----- or "Proc-Type: 4,ENCRYPTED" ``` {% endtab %} {% endtabs %} ### Verify File Permissions {% tabs %} {% tab title="Linux/macOS" %} ```bash # Check permissions ls -la /path/to/server.key ls -la /path/to/server.crt # Private key should be 600 (rw-------) # Certificate can be 644 (rw-r--r--) # Fix if needed chmod 600 /path/to/server.key chmod 644 /path/to/server.crt ``` {% endtab %} {% tab title="Windows PowerShell" %} ```powershell # Check permissions icacls C:\certs\server.key icacls C:\certs\server.crt # Fix if needed - restrict private key to current user only icacls C:\certs\server.key /inheritance:r /grant:r "$($env:USERNAME):(R)" ``` {% endtab %} {% endtabs %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.bindplane.com/how-to-guides/security-and-tls/using-tls/reference/testing-verification.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.