Google Cloud Pub/Sub (Pull)

The Google Cloud Pub/Sub source pulls messages from an existing Pub/Sub subscription and generates logs, metrics, or traces from them. The collector reads messages using the subscription's Pull delivery and uses the message payload encoding to determine the telemetry type.

Note: The OpenTelemetry receiver this source uses is in beta and is subject to change.

Supported Telemetry Types

Platform	Metrics	Logs	Traces
Linux	✓	✓	✓
macOS	✓	✓	✓
Windows	✓	✓	✓
Kubernetes Gateway	✓	✓	✓
OpenShift Gateway	✓	✓	✓

Prerequisites

• A Google Cloud project with the Pub/Sub API enabled.
• An existing Pub/Sub subscription configured with Pull delivery. The source does not create the subscription.
• A service account with the Pub/Sub Subscriber role on the subscription.
• Application Default Credentials available to the collector, typically by pointing GOOGLE_APPLICATION_CREDENTIALS at the service account key file (see Authentication).

Configuration

General

Parameter	Type	Default	Description
Choose Telemetry Type	Telemetry Selector	["Logs","Metrics","Traces"]	Which types of telemetry to receive from the subscription. Required.
Project ID	String		Google Cloud Project ID the Pub/Sub client connects to. Required.
Subscription Name	String		Name of the Pub/Sub subscription to receive messages from. Must already exist. Required.

Advanced

Parameter	Type	Default	Description
Payload Encoding	Extension	none	Payload encoding used as a fallback when the message content-type attribute is not set.1 Required. See Payload encoding and compression.
Payload Compression	Enum	none	Payload compression. Use none for attribute-based detection. Valid values: none, gzip.
Client ID	String	""	Client ID used by Pub/Sub to make load balancing decisions.
User Agent	String	""	User agent used by the Pub/Sub client to connect to the service.
Timeout	Duration	12s	Timeout for all API calls.
Endpoint	String	""	Override of the Pub/Sub endpoint. Leave empty for the default endpoint.
Insecure	Boolean	false	Whether the connection should be insecure. Only takes effect when endpoint is set.

1. Use None for attribute-based detection. Metrics and traces support OTLP Protobuf only. Options other than None and OTLP Protobuf require receiver v0.132.0+.

Flow Control

Flow control parameters require the googlecloudpubsub receiver v0.150.0+. They control how aggressively the server delivers unacknowledged messages.

Parameter	Type	Default	Description
Trigger Ack Batch Duration	Duration	10s	Time between each message acknowledge batch.
Stream Ack Deadline	Duration	60s	Ack deadline used for the stream. Minimum 10 seconds, maximum 600 seconds.
Max Outstanding Messages	Integer	0	Threshold of unacked messages at which the server stops sending more messages. Specify 0 for no limit.
Max Outstanding Bytes	Integer	0	Threshold of unacked bytes at which the server stops sending more messages. Specify 0 for no limit.

Authentication

The collector authenticates to Pub/Sub using Application Default Credentials. The service account must have the Pub/Sub Subscriber role on the target subscription, and the subscription must use Pull delivery.

Point the GOOGLE_APPLICATION_CREDENTIALS environment variable at the service account key file for the collector process. How you set the variable depends on the collector's platform.

Linux

Use a Systemd override:

sudo systemctl edit observiq-otel-collector

Add the environment variable, then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart observiq-otel-collector

Windows

Start PowerShell as administrator and set the variable on the service:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\observiq-otel-collector" /v Environment /t REG_MULTI_SZ /d "GOOGLE_APPLICATION_CREDENTIALS=<path to key file>" /f

Then restart the service:

Restart-Service observiq-otel-collector

macOS

Add GOOGLE_APPLICATION_CREDENTIALS to the EnvironmentVariables dict in /Library/LaunchDaemons/com.observiq.collector.plist (other values shown for context):

<key>EnvironmentVariables</key>
<dict>
    <key>GOOGLE_APPLICATION_CREDENTIALS</key>
    <string><path to key file></string>
    <key>OIQ_OTEL_COLLECTOR_HOME</key>
    <string>/opt/observiq-otel-collector/</string>
    <key>OIQ_OTEL_COLLECTOR_STORAGE</key>
    <string>/opt/observiq-otel-collector/storage</string>
</dict>

Then reload the collector:

sudo launchctl unload /Library/LaunchDaemons/com.observiq.collector.plist
sudo launchctl load /Library/LaunchDaemons/com.observiq.collector.plist

Payload encoding and compression

The receiver inspects message attributes to decide how to interpret each Pub/Sub message:

• content-type controls how the message content is interpreted. If it is set to application/protobuf, a ce-type attribute is expected to identify the telemetry type, one of org.opentelemetry.otlp.traces.v1, org.opentelemetry.otlp.metrics.v1, or org.opentelemetry.otlp.logs.v1.
• content-encoding controls the message compression.

When these attributes are present, attribute-based detection applies and the encoding_extension and compression parameters are ignored, so set both to none / None. When the attributes are absent, the receiver falls back to the encoding_extension and compression parameters.

Metrics and traces support OTLP Protobuf only. Logs can be OTLP JSON, OTLP Protobuf, or raw text. Encoding options other than None and OTLP Protobuf require receiver v0.132.0+.

Example message attributes

A publisher sending OTLP Protobuf logs with gzip compression would set:

content-type: application/protobuf
ce-type: org.opentelemetry.otlp.logs.v1
content-encoding: gzip

With these attributes set, configure the source with encoding_extension = None and compression = none.

Example Configuration

Standalone Source

This example pulls logs from a subscription. The encoding attributes are not present on the message data, so a text encoding extension is used to decode the payload and compression determines compression.

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: googlecloudpubsub
  name: googlecloudpubsub
spec:
  type: googlecloudpubsub_v2
  parameters:
    - name: telemetry_types
      value:
        - Logs
    - name: project
      value: 'my-gcp-project'
    - name: subscription
      value: 'my-subscription'
    - name: compression
      value: 'none'
    - name: encoding_extension
      value:
        disabled: false
        type: text_encoding_unmarshaling
        parameters:
          - name: text_encoding
            value: utf8
          - name: text_unmarshaling_separator
            value: \n?\r

Configuration Tips

Encoding and compression

• Prefer attribute-based detection: set content-type and content-encoding on the publisher and leave encoding_extension and compression set to None / none.
• If your publisher cannot set attributes, set encoding_extension and compression to match the actual payload format. Remember that metrics and traces must be OTLP Protobuf.

Flow control

• Use max_outstanding_messages and max_outstanding_bytes to cap how much unacknowledged data the collector buffers, which helps control memory on a busy subscription. 0 means no limit.
• Flow control parameters only take effect with googlecloudpubsub receiver v0.150.0+. On older collectors they are ignored.

Troubleshooting

Symptom: No messages are received from the subscription. Solution: Confirm the subscription exists, uses Pull delivery, and that the service account has the Pub/Sub Subscriber role. Verify GOOGLE_APPLICATION_CREDENTIALS points to a valid key file readable by the collector process and that the collector was restarted after the variable was set.

Symptom: Messages arrive but fail to parse, or telemetry of the wrong type is produced. Solution: Check the message encoding. If the publisher sets content-type / ce-type, set encoding_extension to None and compression to none so attribute-based detection is used. If those attributes are absent, set encoding_extension and compression to match the payload. Metrics and traces must be OTLP Protobuf.

Symptom: The collector consumes excessive memory under high message volume. Solution: On a collector running receiver v0.150.0+, set max_outstanding_messages and/or max_outstanding_bytes to bound the unacknowledged backlog the server delivers.

Related Resources

• Google Cloud Pub/Sub documentation
• Create a Pub/Sub subscription
• Pub/Sub flow control
• Pub/Sub access control roles