Using Logstash with Bindplane

Bindplane and the Bindplane Collector can be used to collect data from your Logstash agents. This allows you to start taking advantage of Bindplane without the need to re-instrument your collectors at the edge.

Step 1: Update your output stanza of the logstash conf.d files on your Logstash agents

Caveats

Bindplane expects the output from Logstash to be in JSON format. It depends on the codec => json_lines configuration, such as in the examples below to work as expected.

Example output stanza

This output stanza sends to a Bindplane Collector installed on a host with the IP 10.10.1.5, and configured to listen on port 2255 (the default).

output {
  tcp {
    codec => json_lines # this is required otherwise it will send everything in a single line
    host => "10.10.1.5"
    port => 2255
  }
}

Below are a pair of sample logstash conf.d files. After adding these, or modifying the output stanza of existing ones, restart the logstash service.

Config for collecting from /var/log files using the logstash file plugin

# Sample Logstash configuration for creating a simple
# file -> logstash -> otel
input {
  file {
    id => "logstash_var_log_plugin"
    path => ["/var/log/messages", "/var/log/*", "/var/log/**/*"]
    exclude => ["*.gz", "*.journal", "daemon.log", "logstash*.log"]
  }
}

output {
  tcp {
    codec => json_lines # this is required otherwise it will send everything in a single line
    host => "localhost"
    port => 2255
  }
}

Config for collecting from a json formatted log file using the logstash beats plugin

# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.

# filebeats -> logstash -> otel

input {
  beats {
    id => "logstash_beats_otel"
    port => "5044"
  }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
  tcp {
    codec => json_lines # this is required otherwise it will send everything in a single line
    host => "localhost"
    port => 2255
  }
}

Step 2: Deploy a Bindplane Collector as a Gateway

This is the agent you will be routing data through and is what will be managed by Bindplane. In a production environment, this is likely to be a fleet of agents behind a load balancer. See our Collector Sizing and Scaling docs for more details on determining your collector architecture.

Step 3: Build the Configuration

1. Create a new configuration

2. Add the Logstash Source and configure it to receive from your Logstash agents (as shown below)

1. Add a destination of your choice and configure it.

Step 4: Transform the Data

Once you have verified data is flowing through the Bindplane Collector to your destination without issue, you can now start re-routing data to different destinations and inserting processors into your pipeline to reduce the amount of data you are sending.