Kubernetes Postgres Migration

As your Bindplane environment grows in size and importance, it can be desirable to migrate Bindplane's backend to Postgres. Using Postgres allows Bindplane to operate in High Availability.

Prerequisites

The following prerequisites must be met before performing the migration.

CLI

WARNING

Your CLI version should match your control plane version.

The migration requires the Bindplane CLI. Make sure you have a profile configured.

Example profile configuration:

bindplane profile set "bindplane-single" \
  --remote-url "http://bindplane.corp.net" \
  --username "username" \
  --password "password"

bindplane profile use "bindplane-single"

Credentials

The migration process will involve exporting and importing resources using the Bindplane CLI. The export process will not return sensitive data, such as credentials. You must have access to all credentials used by your resources, such as usernames and passwords, API keys, etc.

Deprecated Resources

When resources are deprecated by Bindplane, they are removed unless they are in use. When migrating from Bolt store to Postgres, any deprecated resources will need to be replaced. Generally, a resource is deprecated and replaced by a newer version if a breaking change is made.

Using the Bindplane CLI and jq, check to see if any of your resource types are deprecated.

bindplane get source-types -o json | jq 'map({ name: .metadata.name, deprecated: .metadata.deprecated })'
bindplane get processor-types -o json | jq 'map({ name: .metadata.name, deprecated: .metadata.deprecated })'
bindplane get destination-types -o json | jq 'map({ name: .metadata.name, deprecated: .metadata.deprecated })'

Example output. The parse_severity processor is deprecated. The delete_empty_values processor is not deprecated.

[
  {
    "name": "parse_severity",
    "deprecated": true
  },
  {
    "name": "delete_empty_values",
    "deprecated": null
  }
]

If there are deprecated resource types, it indicates that you have a configuration or resource library component that is deprecated. Remove the component from your configurations and delete it from the resource library, if it is present there.

Secret Key

Note the secret key. You can find your secret key with the bindplane get account command.

Example:

// cspell:ignore 01GR1PHH0K42NW2M0M868XTXDP 21GR1PHH0KB534010JGB9CXYNX

ID                        	NAME        SECRETKEY
01GR1PHH0K42NW2M0M868XTXDP	my-account  21GR1PHH0KB534010JGB9CXYNX

You will need your secret key during the migration process.

Postgres

Postgres should be pre-provisioned and include a database named bindplane or something similar. There should be a user with full access to the database named bindplane. The Bindplane Helm chart does not deploy or manage the Postgres instance.

Max Connections

Make a note of the maximum connections allowed to Postgres. When configuring Bindplane, you can set the maximum number of connections per Bindplane pod.

Prometheus

During the migration, a new Prometheus instance will be deployed by the Helm chart. This means you will lose existing Prometheus data. Prometheus is used by Bindplane to track collector throughput and data reduction metrics. It is not considered critical data when performing the migration.

If you manage your own Prometheus instance, outside of the Helm chart, you will not lose data.

Migration Steps

During the migration, users mustn't modify Bindplane resources. Doing so will cause the exported resources to become out of sync.

1. Export Resources

WARNING

Make sure there are no pending configuration changes. When exporting a configuration, the pending version will be exported if changes are present.

Export resources to a directory called resources. Avoid modifying resources after exporting.

mkdir resources
bindplane get destination -o yaml --export > resources/destination.yaml
bindplane get processor -o yaml --export > resources/processor.yaml
bindplane get source -o yaml --export > resources/source.yaml
bindplane get configuration -o yaml --export > resources/configuration.yaml

1.1 Sensitive Values

Using a text editor, find all instances of (sensitive) in your resource files. Make a note of each resource that contains a sensitive value. When the resources are imported, you will need to update them with the correct value.

2. Deploy High Availability Bindplane

WARNING

Make sure the new Bindplane deployment uses the same image tag as your current deployment.

2.1 Helm Values

Copy your Helm values file to a new file, values-ha.yaml or backup the existing values file.

Make the following changes to the HA values file.

Add the replica top level key, and set it to 3

replicas: 3

Set backend.type to postgres

backend:
  type: postgres

Configure backend.postgres with the values that are correct for your Postgres instance.

backend:
  postgres:
    host: postgres.corp.net
    port: 5432
    database: bindplane
    sslmode: disable # TLS is supported with additional configuration.
    username: bindplane
    password: bindplane-password
    maxConnections: 20 # (replicas * maxConnections) should not exceed your Postgres instance's max connection

Update the top-level resources section to reflect multiple pods. Follow the High Availability Sizing documentation for details.

resources:
  requests:
    memory: 4192Mi
    cpu: 2000m
  limits:
    memory: 4192Mi

If using the Helm-managed Prometheus, ensure its resources and capacity are increased if you expect a high collector count.

prometheus:
  resources:
    requests:
      memory: 2048Mi
      cpu: 1000m
    limits:
      memory: 2048Mi
      cpu: 1000m
  storage:
    volumeSize: 120Gi

Enable the NATS Event Bus.

eventbus:
  type: 'nats'

Add the top-level nats configuration. This configuration is not nested under eventbus.

nats:
  resources:
    requests:
      memory: 500Mi
      cpu: 500m
    limits:
      memory: 500Mi

Update config.server_url and config.remote_url to use a "staging" hostname. During the migration, Bindplane will need to be accessed over a temporary hostname to allow the existing deployment to continue functioning.

config:
  server_url: http://bindplane-stage.corp.net:3001
  remote_url: ws://bindplane-stage.corp.net:3001

2.2 Helm Deployment

Deploy the new Bindplane HA environment using Helm. Make sure to use a new name.

WARNING

Chart version 1.12.0 or newer is required for NATS event bus.

helm repo add bindplane \
    https://observiq.github.io/bindplane-op-helm
helm repo update
helm search repo

helm install --values values-ha.yaml "bindplane-ha" bindplane/bindplane

Once deployed, the old and new HA deployments will be present side by side, if deployed to the same namespace. It is expected that some of the pods will have restarted during the initial deployment.

NAME                                            READY   STATUS    RESTARTS      AGE
bindplane-0                                     1/1     Running   0             25m
bindplane-ha-79c77f7f54-4qgb5                   1/1     Running   1 (14s ago)   16s
bindplane-ha-79c77f7f54-ts62v                   1/1     Running   1 (14s ago)   17s
bindplane-ha-79c77f7f54-z6p98                   1/1     Running   1 (14s ago)   16s
bindplane-ha-jobs-5cfcb4dd67-h7k2m              1/1     Running   0             17s
bindplane-ha-nats-0                             1/1     Running   0             17s
bindplane-ha-nats-1                             1/1     Running   0             16s
bindplane-ha-nats-2                             1/1     Running   0             16s
bindplane-ha-prometheus-0                       1/1     Running   0             17s
bindplane-ha-transform-agent-849c69fd96-djw4c   1/1     Running   0             17s
bindplane-prometheus-0                          1/1     Running   0             25m
bindplane-transform-agent-6546748c6f-jbwbv      1/1     Running   0             25m

2.3 Ensure Access

Make sure you can access the new environment using your existing solution (NGINX, Istio Gateway, etc). This may require you to create an additional ingress configuration.

It is important to keep the current ingress solution routing to your old Bindplane deployment, to prevent downtime. Access to the new HA environment should be handled using a temporary solution, allowing you to import resources.

3. Configure Project

Sign into Bindplane for the first time, and follow the onboarding prompts.

3.2 Configure CLI

Create a new CLI profile for accessing the new environment.

bindplane profile set "bindplane-ha" \
  --remote-url "http://bindplane-stage.corp.net" \
  --username "username" \
  --password "password"

bindplane profile use "bindplane-ha"

Ensure the CLI profile is working by issuing a get agent command. The command should not return any collectors. If it does, you might be using the profile for the old Bindplane instance.

3.3 Update Project Secret Key

Update the new project's secret key with the secret key used by the old Bindplane deployment. This will allow collectors to connect to the new Bindplane instance and project.

bindplane get account -o yaml --export > resources/ha-account.yaml

It will look similar to this.

// cspell:ignore HTRNREDRKFXNTYDJD HTRNREX

---
apiVersion: bindplane.observiq.com/v1
kind: Account
metadata:
  id: 01J2HTRNREDRKFXNTYDJD7S4MP
  displayName: bindplane-ha
spec:
  secretKey: 01J2HTRNREX2H93C9TZV0TEF1J
  alternateSecretKeys: []

Update spec.secretKey in ha-account.yaml to match the secret key from the original environment.

apiVersion: bindplane.observiq.com/v1
kind: Account
metadata:
  id: 01J2HTRNREDRKFXNTYDJD7S4MP
  displayName: bindplane-ha
spec:
  # Secret key from old env
  secretKey: 21GR1PHH0KB534010JGB9CXYNX # Orig: 01J2HTRNREX2H93C9TZV0TEF1J
  alternateSecretKeys: []

Update the account using the apply command.

bindplane apply -f resources/ha-account.yaml

Check if it worked: bindplane get account. The output should show the original secret key. You can also check the UI.

4 Import Resources

4.1 Import

Import your resources using the apply command.

bindplane apply -f resources/destination.yaml
bindplane apply -f resources/processor.yaml
bindplane apply -f resources/source.yaml
bindplane apply -f resources/configuration.yaml

4.2 Update Sensitive Values

Find all components that use sensitive values, and update them. The resources exported from the previous Bindplane instance will not contain sensitive values such as usernames and passwords or API keys.

4.3 Rollout Configurations

Trigger a rollout for all configurations to ensure they will be pushed to collectors during the cut-over to the HA instance.

bindplane get config | grep -v NAME | awk '{print $1}' | xargs -I{} bindplane rollout start {}

5 Cut Over Access

Now that the new Bindplane instance has had its secret key configured and resources imported, it is ready to accept connections from collectors.

5.1 Helm Configuration

Update config.server_url and config.remote_url to your primary hostname, the hostname used by the old Bindplane instance. Use Helm to deploy the change.

5.2

Update your ingress solution to point to the new Bindplane HA deployment.

Note that collectors will likely remain connected to the old Bindplane instance until one of two things happens:

The collector(s) are restarted
The old Bindplane instance is restarted

Restart the old Bindplane StatefulSet pod to force collectors to reconnect, allowing them to connect to new HA system.

6 Verify Functionality

Ensure resource utilization is acceptable
Ensure collectors are connecting
Ensure Recent Telemetry is working
Ensure Live Preview is working
Ensure configuration rollout can be triggered and finished

6.1 Abort

If the new system is not working correctly or appears unstable, revert your ingress change to point to the old Bindplane instance. Restart all Bindplane HA pods to force collectors to reconnect to the old Bindplane system.

7 Finish and Cleanup

At this point, you have successfully migrated to Bindplane High Availability with NATS and Postgres.

You can safely and optionally remove the old Bindplane instance using helm uninstall. The New Bindplane deployment does not rely on any resources from the old Bindplane instance.

PreviousKubernetes Cluster Name Detection NextGoogle Kubernetes Engine Gateway Collector Load Balancer

Last updated 1 month ago

Was this helpful?

Prerequisites

CLI

Credentials

Deprecated Resources

Secret Key

Postgres

Max Connections

Prometheus

Migration Steps

1. Export Resources

1.1 Sensitive Values

2. Deploy High Availability Bindplane

2.1 Helm Values

2.2 Helm Deployment

2.3 Ensure Access

3. Configure Project

3.1 Initial Sign In

3.2 Configure CLI

3.3 Update Project Secret Key

4 Import Resources

4.1 Import

4.2 Update Sensitive Values

4.3 Rollout Configurations

5 Cut Over Access

5.1 Helm Configuration

5.2

6 Verify Functionality

6.1 Abort

7 Finish and Cleanup