Google Cloud Storage Event

The Google Cloud Storage Event source consumes Google Cloud Storage object notifications delivered to a Pub/Sub subscription. Pub/Sub is only the delivery path; the source reads GCS objects. When a notification arrives, the source downloads the referenced GCS object and reads its contents into the body of a log record.

Version 2. This version exposes the interval settings as durations, so you can enter values like 500ms, 30m, or 1h instead of a raw seconds count. The previous version is deprecated and kept only for existing pipelines. Migrating a config: each interval that was a number of seconds becomes the equivalent duration string — a Poll Interval of 1 becomes 1s, a Maximum Ack Extension of 3600 becomes 1h.

Supported Telemetry Types

Platform	Metrics	Logs	Traces
Linux		✓
Windows		✓
macOS		✓
Kubernetes Cluster		✓
Kubernetes Gateway		✓

Prerequisites

• A Google Cloud project with access to Cloud Storage and Pub/Sub.

• A GCS bucket notification configured to publish object events to a Pub/Sub topic.

• A Pub/Sub subscription on that topic for the collector to consume.

• Credentials for the collector with permission to pull and acknowledge messages from the subscription and to read objects from the bucket. See Application Default Credentials.

• A reachable Redis server if you enable offset storage (optional).

How It Works

1. The source pulls messages from the Pub/Sub subscription that receives GCS object notifications.

2. For each notification, the source downloads the referenced GCS object.

3. The object contents are read into the body of one or more log records.

4. Notifications are deduplicated in two layers, since GCS publishes notifications at-least-once:

   • Batch-level: within a single pull response, duplicates keyed by bucket, object, and generation are acknowledged immediately.

   • Cross-batch: a time-bounded tracker (configured by Deduplication TTL) catches sequential duplicates that arrive in separate pull responses.

5. When Redis offset storage is enabled, the read position into an object is persisted so processing resumes after a collector restart, allowing large objects to be handled without exhausting memory.

Configuration

General

Parameter	Type	Required	Default	Description
Project ID	String	Yes		The Google Cloud project ID that contains the Pub/Sub subscription.
Subscription ID	String	Yes		The Pub/Sub subscription ID that receives GCS event notifications.
Authentication Method	Enum	No	auto	How the collector authenticates to Google Cloud. auto uses Application Default Credentials. file reads a service account credential file from the collector system.
Credentials File	String	Yes1		Path to a Google Service Account credential file on the collector system. The collector's runtime user must have permission to read it.

1. Shown and required only when Authentication Method is file.

Advanced

Parameter	Type	Required	Default	Description
Workers	Integer	No	5	The number of concurrent workers used to process events.
Maximum Ack Extension	Duration	No	1h	The maximum total time the receiver will extend the ack deadline for a message being processed. After this duration the message becomes eligible for redelivery. Accepts a duration such as 30s, 5m, or 1h.
Poll Interval	Duration	No	1s	The interval the receiver waits between Pub/Sub pulls when no messages are found. When messages are found, the next pull is immediate. Accepts a duration such as 500ms, 1s, or 5s.
Deduplication TTL	Duration	No	5m	How long to remember recently-processed objects for cross-batch deduplication. GCS publishes notifications at-least-once, so duplicates may arrive seconds apart. Accepts a duration such as 30s, 5m, or 1h.
Maximum Log Size	Integer	Yes	1048576	The maximum size in bytes for a single log record. Logs exceeding this size are split into chunks.
Maximum Logs Emitted	Integer	No	1000	The maximum number of log records to emit in a batch. A higher number results in fewer batches but more memory.
Emit Raw Logs	Boolean	No	false	When enabled, the original text is emitted as the log body instead of a parsed structure. Format detection still runs, so content the collector cannot decode is sent to the dead-letter queue.
Include Original Log Record	Boolean	No	true	When enabled, the original text is also written to the log record's original field.
Filter by Bucket Name	CodeBlock	No		When set, the source only emits logs for bucket names that match the specified regex.
Filter by Object Name	CodeBlock	No		When set, the source only emits logs for object names that match the specified regex.

Redis Offset Storage

Offset storage persists the read position into each object to Redis so processing resumes after a collector restart. These parameters apply only when Enable Redis Offset Storage is on.

Parameter	Type	Required	Default	Description
Enable Redis Offset Storage	Boolean	No	false	When enabled, the current position into an object is saved to Redis, and reading resumes from where it left off after a collector restart.
Redis Hostname	String	Yes1	localhost	The hostname or IP address of the Redis server used for offset storage.
Redis Port	Integer	Yes1	6379	The port number of the Redis server used for offset storage. Range 1–65535.
Redis Password	String	No		The password for the Redis server used for offset storage.
Redis Database	Integer	Yes1	0	The Redis database number to use for offset storage.
Redis Expiration	Duration	Yes1	0s	The expiration time for offset storage in Redis. Accepts a duration such as 30s, 5m, or 1h; 0s disables expiration.
Enable Redis TLS	Boolean	Yes1	false	Whether to enable TLS for the Redis connection.
Redis TLS CA File	String	No		The path to the CA file for the Redis TLS connection.
Redis TLS Cert File	String	No		The path to the client certificate file for the Redis TLS connection.
Redis TLS Key File	String	No		The path to the client key file for the Redis TLS connection.

1. Shown and required only when Enable Redis Offset Storage is on.

Examples

File authentication with tuned intervals

File-based authentication and the Advanced section, with Credentials File set and the interval durations configured.

Example Configuration

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: google_cloud_storage_event_v2
  name: google_cloud_storage_event_v2
spec:
  type: google_cloud_storage_event_v2
  parameters:
    - name: project_id
      value: 'my-gcp-project'
    - name: subscription_id
      value: 'gcs-events-sub'
    - name: workers
      value: '5'
    - name: dedup_ttl
      value: '5m'
    - name: max_log_size
      value: '1048576'

Configuration Tips

Authentication

• The source uses Application Default Credentials by default (Authentication Method auto). For a collector that does not run on GCP infrastructure, and so has no Application Default Credentials, set Authentication Method to file and point Credentials File at a service account key the collector's runtime user can read.

Filtering objects

• Use Filter by Bucket Name and Filter by Object Name to restrict processing to a subset of buckets or object keys. Both accept a regular expression and are evaluated against the bucket and object names from the notification.

• Filtering at the source reduces downloads and downstream volume, so apply it when a single subscription receives notifications for objects you do not want to ingest.

Handling large objects

• For large objects, enable Redis offset storage and point it at a Redis server. The source persists the read position so a collector restart resumes mid-object instead of reprocessing or skipping data.

• Tune Maximum Log Size and Maximum Logs Emitted together. Larger values reduce batch count but increase memory use per batch.

Deduplication

• GCS notification delivery is at-least-once. Keep Deduplication TTL at or above the interval over which you expect duplicate notifications to arrive so cross-batch deduplication catches them.

Troubleshooting

Symptom: No logs arrive even though objects are being written to the bucket. Solution: Confirm a GCS bucket notification is publishing to the Pub/Sub topic and that Subscription ID matches a subscription on that topic. Verify the collector's credentials can pull and acknowledge messages on the subscription.

Symptom: Messages are received but objects fail to download. Solution: Ensure the collector's credentials have read access to the GCS bucket. A permission denied (403) or not found (404) on the object causes the message to be nacked for redelivery or dead-letter processing.

Symptom: The same object is ingested more than once. Solution: Increase Deduplication TTL so the cross-batch deduplication window covers the interval over which GCS redelivers notifications. Persistent duplicates beyond the window are expected under at-least-once delivery.

Symptom: A large object is reprocessed from the start after a collector restart. Solution: Enable Redis offset storage and configure a reachable Redis server so the read position is persisted across restarts.

Related Resources

• Cloud Storage Pub/Sub notifications

• Create a Pub/Sub subscription

• Application Default Credentials