Switching Queues

This page explains how you can safely configure and edit your persistent queue without losing data.

How to Prevent Data Loss When Switching Queues?

The principal danger when changing a destination's persistent queue configuration is data loss.

Dangerous Edits

If a destination has a specific persistent queue configured, persists data to that queue, and then is restarted with a new persistent queue configuration before it is able to dequeue the persisted data, that data will be trapped in the old queue forever.

Changing the persistent queue storage type

Changing from one persistent queue storage solution to another will result in a new queue being made for the new solution, with the old queue untouched.

Changing the storage directory

Changing the persistent queue directory will result in a new queue being stored in the new location, with the old queue untouched.

Safe Edits

This describes the steps to modify a destination's persistent queue configuration safely without data loss. If data loss is not a significant concern, the queue configuration can be modified however desired.

When the persistent queue is empty for a destination, its settings can be safely modified directly in the destination. Queue state can be monitored through Internal Telemetry.

When the persistent queue is not empty, safe queue modification requires the Custom source and a Version 2 configuration.

Let's say that you have a simple configuration sending TCP logs to an OTLP destination using a Badger persistent queue.

Next, let's say that:

• this configuration is rolled out to an agent

• the Badger persistent queue is not empty

• we'd like to change the persistent queue to be a Pebble persistent queue.

In order to achieve this, we're going to have to alter our configuration to have two pipelines - one that drains the existing Badger persistent queue, and one which reroutes any new data to our new Pebble persistent queue. This can be achieved with a few easy steps:

First, we're going to duplicate our existing OTLP destination, but tweak the duplicate to use the Pebble persistent queue instead. Add this duplicate as a second destination to the configuration.

Second, we're going to add a new Custom source that won't send any logs.

Third, we're going to reroute all existing traffic away from the Badger destination, to the Pebble destination.

Finally, we're going to add a route from the Custom receiver to the Badger destination.

Once this is done, the configuration can be rolled out to the agent. The top pipeline will ensure that any new TCP Logs received are routed to the OTLP destination using Pebble as a persistent queue storage solution. The bottom pipeline will ensure that no new data will be sent to the Badger persistent queue, and the existing data in the Badger persistent queue will be sent to the OTLP destination.

Once the Badger persistent queue has been drained (again, this can be monitored via Internal Telemetry), the bottom pipeline can be removed from the configuration, and it can be rolled out for a final time to the agent.