Snowflake

Description

This destination can send logs, metrics, and traces to Snowflake, a cloud data warehouse service.

Supported Types

Logs	Metrics	Traces	Bindplane Collector
✓	✓	✓	v1.45.0+

Prerequisites

• Snowflake data warehouse

• Snowflake user with appropriate privileges

The following guide will detail how to get a user initialized in Snowflake that can be used with this destination. It is meant to get telemetry flowing with limited time spent configuring. If you'd like to take an alternative approach, check out the exporter documentation on this topic here.

Snowflake has a variety of ways to connect to it, but these instructions will be tailored for "Classic Console" as all accounts have access to it.

Before starting, log in to Classic Console using a user that has access to the ACCOUNTADMIN role or another role in you Snowflake account that has permission to grant privileges and create users. If the default role is not the required one, then you'll need to assume that role using this SQL command (replace the role as needed):

ASSUME ROLE "ACCOUNTADMIN";

These instructions will grant privileges to one of the default roles Snowflake is initialized with, SYSADMIN. If you want to grant privileges to a different role then just switch out SYSADMIN for your role in the SQL commands.

1. Grant Warehouse Usage

First, we need to grant the USAGE privilege to the SYSADMIN role on the data warehouse telemetry data will be stored in. Run this SQL command next (replace TEST with your warehouse name):

GRANT USAGE ON WAREHOUSE "TEST" TO ROLE "SYSADMIN";

2. Grant Create Database Privilege

Next the SYSADMIN role needs to be granted the ability to create databases in the Snowflake account. Run the following SQL to do so:

GRANT CREATE DATABASE ON ACCOUNT TO ROLE "SYSADMIN";

3. Create New User For Bindplane

Now a new user needs to be created that the Bindplane Collector can login as. The user should also have the default role assigned as SYSADMIN, although it isn't necessary.

Note: If the default role is not assigned, then the exporter will need to be configured with the correct role to work.

Remember the login name and password you use and configure the destination with these values. Replace the user, password, and login name in the following SQL to match yours:

CREATE USER BP_AGENT PASSWORD="password" LOGIN_NAME="BP_AGENT" DEFAULT_ROLE="SYSADMIN";

4. Grant Privilege to SYSADMIN Role

Even though the default role was set as SYSADMIN we still need to grant the new account permission to it. This can be done using the next SQL command (replace user as needed):

GRANT ROLE "SYSADMIN" TO USER BP_AGENT;

Now we have a Snowflake user with the correct permissions to be able to create a database, schemas, and tables and also use the configured warehouse to store telemetry data in.

Configuration

Field	Description
Choose Telemetry Type	The kinds of telemetry to send to Snowflake.
Account Identifier	The account identifier for the Snowflake account that data should be sent to.
Warehouse	THe Snowflake warehouse that telemetry data should be stored in.
Username	Username the exporter will use to authenticate with Snowflake.
Password	Password the exporter will use to authenticate with Snowflake.
Database	The Snowflake database that telemetry schemas will be stored in. Will be created if it doesn't exist.
Log Schema	The schema that the log table will be stored in. Will be created if it doesn't exist.
Log Table	The table that logs will be stored in. Will be created if it doesn't exist.
Metric Schema	The schema that the metric tables will be stored in. Will be created if it doesn't exist.
Metric Table	The prefix used for metric tables. Tables are created if they don't exist. See this exporter documentation for more.
Trace Schema	The schema that the trace table will be stored in. Will be created if it doesn't exist.
Trace Table	The table that traces will be stored in. Will be created if it doesn't exist.
Role	The Snowflake role the exporter should use. Only required if the default role of the provided credentials does not have correct privileges.
Parameters	Additional optional parameters the exporter should use when connecting to Snowflake. This option is generally not required. See this Snowflake documentation for more.

This destination supports the following retry and queuing settings:

Sending Queue	Persistent Queue	Retry on Failure
✓	✓	✓

Example Configuration

Here we will configure this destination to send all telemetry types to a Snowflake account using the default values for database resources. We'll also configure the sending queue, persistent queue, and retry on failure.

Web Interface

Standalone Destination

apiVersion: bindplane.observiq.com/v1
kind: Destination
metadata:
  id: snowflake
  name: snowflake
spec:
  type: snowflake
  parameters:
    - name: telemetry_types
      value: ['Logs', 'Metrics', 'Traces']
    - name: account_id
      value: 'my_account_id'
    - name: warehouse
      value: 'my_warehouse'
    - name: username
      value: 'my_user'
    - name: password
      value: 'my_pass'
    - name: sending_queue_enabled
      value: true
    - name: persistent_queue_enabled
      value: true
    - name: retry_on_failure_enabled
      value: true