Amazon Managed Prometheus

Description

This Amazon Managed Prometheus destination can be used to send metrics to an Amazon Managed Prometheus (AMP) workspace in AWS.

Supported Types

Prerequisites

Getting the Bindplane Collector authenticated and authorized with AWS requires completing the following steps. Before starting, make sure you are logged in to AWS Console and have permission to create users, create roles, and generate access tokens.

1. Create an AWS User the exporter can use.

To start, head to the Identity and Access Management (IAM) service and under Access management navigate to Users. There should be an option to Create user on this page. This is the user the exporter and Bindplane Collector will be authenticated as, so name it appropriately. This new user will not need access to AWS Management Console, so leave this option unselected. For permissions, you don't need to specify any at this point since the user will be assuming a role with the required permissions. Once the user is created, navigate to its summary page and copy the ARN (we will need it in the next step).

2. Create an AWS Role the exporter can assume.

Now we need to create an AWS Role the exporter will assume. This is what will allow the exporter/Bindplane Collector to send data to the AMP workspace. To do this, head to IAM and under Access management navigate to Roles and select Create role. Under the first step, Select trusted entity, we are defining which AWS resource will be allowed to use this role, which we want to be the user we created in the first step. Select Custom trust policy. We want to edit just the Principal JSON field to contain the ARN copied in the first step. It should look similar to this:

"Principal": {
    "AWS": "arn:aws:iam::123456789:user/Bindplane"
}

The next step in creating the role is Add permissions. Search for "Prometheus" in the search bar and a number of permissions should show up. The exporter only needs AmazonPrometheusRemoteWriteAccess, so select that. In the next area, give the role an appropriate name and description and finish with Create role.

As an alternative to the AWS pre-defined AmazonPrometheusRemoteWriteAccess permission policy, you may also create your own permission policy. To read more, see this AWS documentation. If taking this approach, be sure the exporter will still have write access to the desired AMP Workspace or the Bindplane Collector will fail.

3. Give Bindplane Collector access keys.

Now that the Bindplane Collector will have access to the AMP workspace, it needs access keys so it can access AWS as the user that was created for it in step 1. Head back to IAM and Users and select the user created in step 1. Now select the Security credentials tab and find the Access keys section. For the first step of creating a key, select Other and continue to the next step. Provide an appropriate description and select Create access key. Make note of or download the Access key and Secret access key values as AWS won't show these again.

Now that the access keys have been created, we need to create environment variables for them on the same machine as the Bindplane Collector. The environment variables to add are AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, which correspond to the values retrieved early in this step.

You can modify the collector's environment variables by configuring a systemd override. Run sudo systemctl edit observiq-otel-collector and add the keys:

[Service]
Environment=AWS_ACCESS_KEY_ID=******
Environment=AWS_SECRET_ACCESS_KEY=******

When finished, reload Systemd.

sudo systemctl daemon-reload

Now the Bindplane Collector will be able to authenticate with AWS as the user that was created back in step 1.

For more information on specifying the AWS credentials for the Bindplane Collector, please see this AWS documentation. See this AWS documentation for alternative ways of using AWS environment variables for you specific environment.

4. Retrieve Amazon Managed Prometheus workspace endpoint.

The last pre-requisite step to configuring the Amazon Managed Prometheus destination is retrieving the remote write endpoint. If you haven't already created an Amazon Managed Prometheus workspace, search for "Amazon Prometheus" in the AWS console search bar and initialize one. With a workspace created, navigate to All workspaces inside of the Amazon Prometheus AWS service. Select the workspace you wish to send metrics to. On the summary page, copy the value for Endpoint - remote write URL. This is the endpoint you'll use when configuring the destination.

Configuration

This destination supports the following retry and queuing settings:

Example Configuration

Here we configure the destination to send to AMP by providing the endpoint, region, and arn. Some advanced configuration options we make use of include compression, namespace, max batch size, convert attributes, unit and type suffixes, created metric, and external labels by defining a label called "bp_agent" with a value of "agent1". We also enable TLS, Strict TLS, and Mutual TLS and provide a certificate authority file, client certificate file, and client key file. We also have a remote write queue and Retry on Failure enabled with their respective default configurations.

Web Interface

Standalone Destination

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: aws_managed_prometheus
  name: aws_managed_prometheus
spec:
  type: aws_managed_prometheus
  parameters:
    - name: endpoint
      value: some_aws_amp_workspace_endpoint
    - name: region
      value: us-east-2
    - name: arn
      value: some_aws_role_arn
    - name: compression
      value: snappy
    - name: namespace
      value: bp_amp
    - name: max_batch_size_bytes
      value: 3000000
    - name: enable_resource_to_telemetry_conversion
      value: true
    - name: add_unit_and_type
      value: true
    - name: export_created_metric
      value: true
    - name: external_labels
      value:
        bp_agent: agent1
    - name: enable_tls
      value: true
    - name: strict_tls_verify
      value: true
    - name: ca_file
      value: some_ca_file
    - name: mutual_tls
      value: true
    - name: cert_file
      value: some_cert_file
    - name: key_file
      value: some_key_file
    - name: enable_remote_write_queue
      value: true
    - name: retry_on_failure_enabled
      value: true