Configuration

When operating a self-managed Prometheus instance, Bindplane's server configuration must be updated to connect to the remote Prometheus instance.

Bindplane Configuration

After installing Bindplane, update the configuration file at /etc/bindplane/config.yamlusing the editor of your choice.

• Set prometheus.enableRemote to true

• Set prometheus.host to the IP address or Hostname of your Prometheus server.

prometheus:
  enableRemote: true
  localFolder: /var/lib/bindplane/prometheus
  host: prometheus.c.project.internal
  port: '9090'
  remoteWrite:
    endpoint: /api/v1/write
  auth:
    type: none

Once enableRemote and host are configured, restart the Bindplane server process.

sudo systemctl restart bindplane

At this point, Bindplane is installed and configured to use the remote Prometheus instance.

Security

Prometheus supports several options for security. Basic authentication (Basic auth), Transport Layer Security (TLS), and Mutual TLS (mTLS).

Basic Authentication

Follow the Prometheus Basic Auth Password Hashing documentation to generate a password hash.

Once you have your hash, update /etc/prometheus/web.yml with your basic auth username and password hash.

// cspell:ignore maOicLymWgsIQleRCm604ePbaaavp9cKj3bJUg0IrcVXCHB3terLa

# Example use only: admin:password
basic_auth_users:
  admin: $2b$12$maOicLymWgsIQleRCm604ePbaaavp9cKj3bJUg0IrcVXCHB3terLa

Restart the Prometheus service.

sudo systemctl restart prometheus

Test by making a curl request, without basic auth. You should expect a "401 Unauthorized" response.

curl -v -s localhost:9090/metrics > /dev/null

Test by making a curl request with your username and password.

curl -v -s -u 'admin:password' localhost:9090/metrics > /dev/null

You should expect a "200 OK" response. This will indicate that basic auth is working correctly.

Next, we need to update Bindplane with the new credentials. Edit /etc/bindplane/config.yaml on all of your Bindplane servers.

prometheus:
  enableRemote: true
  localFolder: /var/lib/bindplane/prometheus
  host: prometheus.c.bpcli-dev.internal
  port: '9090'
  remoteWrite:
    endpoint: /api/v1/write
  auth:
    type: basic
    username: admin
    password: password

Restart the Bindplane service.

sudo systemctl restart bindplane

Transport Layer Security (TLS)

Copy the certificate keypair to /etc/prometheus/tls. The example commands assume that you have a certificate key pair in your working directory named prometheus.crt and prometheus.key

sudo mkdir /etc/prometheus/tls

sudo mv prometheus.crt prometheus.key /etc/prometheus/tls

sudo chown -R prometheus:prometheus /etc/prometheus/tls
sudo chmod 0600 \
  /etc/prometheus/tls/prometheus.crt \
  /etc/prometheus/tls/prometheus.key

Server side TLS can be configured by editing the web configuration file at /etc/prometheus/web.yml and configuring the certificate file and private key file paths.

tls_server_config:
  cert_file: /etc/prometheus/tls/prometheus.crt
  key_file: /etc/prometheus/tls/prometheus.key

Restart the Prometheus service.

sudo systemctl restart prometheus

You can test if Prometheus is using TLS by using curl.

curl -kvs https://localhost:9090/metrics > /dev/null

You should expect a "200 OK" response. This will indicate that server side TLS is working correctly.

Next, we need to update Bindplane to use TLS when communicating with Prometheus. On all of your servers, perform the following steps.

Copy the certificate authority to /etc/bindplane/tls. The example commands assume that you have a certificate authority public key named ca.crt in your working directory.

sudo mkdir /etc/bindplane/tls
sudo mv ca.crt /etc/bindplane/tls

sudo chown -R bindplane:bindplane /etc/bindplane/tls
sudo chmod 0600 /etc/bindplane/tls/ca.crt

Edit /etc/bindplane/config.yaml on all of your Bindplane servers and add the tls.tlsCa parameter.

prometheus:
  enableRemote: true
  localFolder: /var/lib/bindplane/prometheus
  host: prometheus.c.bpcli-dev.internal
  port: '9090'
  remoteWrite:
    endpoint: /api/v1/write
  auth:
    type: none
  enableTLS: true
  tls:
    tlsSkipVerify: false
    tlsCa:
      - /etc/bindplane/tls/ca.crt

NOTE

Make sure prometheus.host matches the hostname of the Prometheus server's certificate. If the hostname does not match, you can set prometheus.tls.tlsSkipVerify to true to skip TLS verification. Skipping TLS verification is not recommended in a production environment.

Restart the Bindplane service.

sudo systemctl restart bindplane

Mutual TLS

Copy the certificate keypair and certificate authority to/etc/prometheus/tls. The example commands assume that you have a certificate key pair in your working directory named prometheus.crt and prometheus.key and a certificate authority named ca.crt.

sudo mkdir /etc/prometheus/tls

sudo mv prometheus.crt prometheus.key ca.crt /etc/prometheus/tls

sudo chown -R prometheus:prometheus /etc/prometheus/tls
sudo chmod 0600 \
  /etc/prometheus/tls/prometheus.crt \
  /etc/prometheus/tls/prometheus.key \
  /etc/prometheus/tls/ca.crt

Mutual TLS can be configured by editing the web configuration file at /etc/prometheus/web.yml and configuring the certificate file, private key file paths and certificate authority paths.

tls_server_config:
  client_auth_type: RequireAndVerifyClientCert
  client_ca_file: /etc/prometheus/tls/ca.crt
  cert_file: /etc/prometheus/tls/prometheus.crt
  key_file: /etc/prometheus/tls/prometheus.key

Restart the Prometheus service.

sudo systemctl restart prometheus

You can test if Prometheus is using TLS by using curl on the Prometheus system.

# Sudo is required to read the TLS certificate files
# in /etc/prometheus/tls.
# Replace $(hostname -f) with the hostname that matches
# the prometheus server and certificate.
sudo curl -vs \
  --cacert /etc/prometheus/tls/ca.crt \
  --cert /etc/prometheus/tls/prometheus.crt \
  --key /etc/prometheus/tls/prometheus.key \
  "https://$(hostname -f):9090/metrics" > /dev/null

You should expect a "200 OK" response. This will indicate that mutual TLS is working correctly.

Next, we need to update Bindplane to use mutual TLS when communicating with Prometheus. On all of your servers, perform the following steps.

Copy the certificate authority and client keypair to /etc/bindplane/tls. The example commands assume that you have a certificate key pair in your working directory named bindplane.crt and bindplane.key and a certficate authority named ca.crt.

sudo mkdir /etc/bindplane/tls
sudo mv bindplane.crt bindplane.key ca.crt /etc/bindplane/tls

sudo chown -R bindplane:bindplane /etc/bindplane/tls
sudo chmod 0600 \
  /etc/bindplane/tls/bindplane.crt \
  /etc/bindplane/tls/bindplane.key \
  /etc/bindplane/tls/ca.crt

Edit /etc/bindplane/config.yaml on all of your Bindplane servers and add the tls parameters.

prometheus:
  enableRemote: true
  localFolder: /var/lib/bindplane/prometheus
  host: prometheus.c.bpcli-dev.internal
  port: '9090'
  remoteWrite:
    endpoint: /api/v1/write
  auth:
    type: none
  enableTLS: true
  tls:
    tlsSkipVerify: false
    tlsCa:
      - /etc/bindplane/tls/ca.crt
    tlsCert: /etc/bindplane/tls/bindplane.crt
    tlsKey: /etc/bindplane/tls/bindplane.key

NOTE

Make sure prometheus.host matches the hostname of the Prometheus server's certificate. If the hostname does not match, you can set prometheus.tls.tlsSkipVerify to true to skip TLS verification. Skipping TLS verification is not recommended in a production environment.

Restart the Bindplane service.

sudo systemctl restart bindplane