Google Cloud Storage Rehydration

Supported Types

Metrics	Logs	Traces
✓	✓	✓

How It Works

1. This source rehydrates data previously stored by the Google Cloud Storage Destination.

2. It will process both uncompressed JSON objects and objects compressed with gzip.

3. You can authenticate to Google Cloud using the provided credentials, credentials_file, or by using Application Default Credentials.

4. Your authentication credentials must have the Storage Admin permission to read and delete objects.

Notes

This is not a traditional source that continually produces data. Instead, it rehydrates all objects found within a specified time range. Once all objects in that time range have been rehydrated, the source will stop producing data.

Configuration

Parameter	Type	Default	Description
telemetry_types*	telemetrySelector	Logs, Metrics, Traces	Specifies which types of telemetry to rehydrate.
bucket_name*	string	""	The name of the bucket to rehydrate from.
project_id	string	""	The ID of the Google Cloud project the bucket belongs to. Will be read from credentials if not configured.
auth_type	enum	auto	The method used for authenticating to Google Cloud. Valid values are "auto", "json", or "file".
credentials	string	""	JSON value from a Google Service Account credential file. Required if auth_type is "json".
credentials_file	string	""	Path to a Google Service Account credential file. Required if auth_type is "file".
starting_time*	dateTime	""	The UTC start time for rehydration. Must be in the format "YYYY-MM-DDTHH:MM".
ending_time*	dateTime	""	The UTC end time for rehydration. Must be in the format "YYYY-MM-DDTHH:MM".
folder_name	string	""	Restricts rehydration to objects in a specific folder within the bucket.
batch_size	int	30	The number of objects to download at once. This impacts performance by controlling the number of concurrent object downloads.
delete_on_read	bool	false	If true, objects will be deleted after being rehydrated.
storage_enable	bool	true	Enable to specify a storage extension for rehydration progress.
storage_directory	string	$OIQ_OTEL_COLLECTOR_HOME/storage	Directory for storing rehydration state. Useful for maintaining state and resuming operations after disruptions.

*required field

Example Configurations

Basic Configuration

This configuration authenticates using Application Default Credentials and rehydrates data in the specified bucket, folder, and time range.

Web Interface

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: google_cloud_storage_rehydration
  name: google_cloud_storage_rehydration
spec:
  type: google_cloud_storage_rehydration
  parameters:
    - name: telemetry_types
      value: ['Logs', 'Metrics', 'Traces']
    - name: bucket_name
      value: 'my-bucket'
    - name: auth_type
      value: 'auto'
    - name: starting_time
      value: '2025-03-03T16:00'
    - name: ending_time
      value: '2025-03-03T17:00'
    - name: folder_name
      value: 'my-folder-name'
    - name: batch_size
      value: 30
    - name: storage_enable
      value: false

Complete Configuration

This configuration demonstrates all available options for the Google Cloud Storage Rehydration source, including authentication, storage settings, and delete on read functionality.

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: google_cloud_storage_rehydration
  name: google_cloud_storage_rehydration
spec:
  type: google_cloud_storage_rehydration
  parameters:
    - name: telemetry_types
      value: ['Logs', 'Metrics', 'Traces']
    - name: bucket_name
      value: 'my-bucket'
    - name: project_id
      value: 'my-project'
    - name: auth_type
      value: 'file'
    - name: credentials_file
      value: '/path/to/googlecloud/credentials/file'
    - name: starting_time
      value: '2025-03-03T16:00'
    - name: ending_time
      value: '2025-03-03T17:00'
    - name: folder_name
      value: 'my-folder'
    - name: batch_size
      value: 30
    - name: delete_on_read
      value: true
    - name: storage_enable
      value: true
    - name: storage_directory
      value: '/custom/storage/path'