Google Cloud Pub/Sub (Push)

Description

Receive messages from a Google Cloud PubSub topic and generate logs from the result.

IMPORTANT

The OpenTelemetry receiver this source uses is in development and is subject to change.

Supported Platforms

Platform	Supported
Linux	✓
Windows	✓
macOS	✓

Configuration Table

Parameter	Type	Default	Description
listen_address	string	0.0.0.0	The IP address to listen on.
listen_port	int	8090	TCP port to receive Google Cloud Pub/Sub requests.
encoding_extension	extension	type: googlecloudlogentry_encoding parameters: - name: cloud_logging_json_handling value: json - name: cloud_logging_proto_handling value: json	The extension to use for parsing the payload.
enable_tls	bool	false	Whether or not to use TLS.
cert_file	string		Path to the TLS certificate file for the server.
key_file	string		Path to the TLS private key file for the server.
mutual_tls	bool	false	Whether or not to require client TLS authentication (mTLS).
ca_file	string		Path to the certificate authority file for authenticating client certificates.
include_metadata	bool	false	Propagates the incoming connection's metadata to downstream consumers.
enable_cors	bool	false	Enable Cross-Origin Resource Sharing support for HTTP requests.
cors_allowed_origins	strings	[]	A list of origins a cross-domain request can be executed from.
cors_allowed_headers	strings	[]	The allowed headers sets what headers will be allowed in CORS requests.
cors_max_age	int	0	Indicates how long (in seconds) the results of a preflight request can be cached.

Authentication

The service account used by the collector should have the Pub/Sub Subscriber permission to read from the subscription. Additionally, make sure the subscription channel being received from is configured with Push delivery.

Authentication is handled via the GOOGLE_APPLICATION_CREDENTIALS environment variable. This should point to a file containing the service account key for the project.

To set this for the collector, you need to modify the service file depending on your OS.

Linux

Use a Systemd override:

sudo systemctl edit observiq-otel-collector

Then run the following command to reload the systemd configuration:

sudo systemctl daemon-reload
sudo systemctl restart observiq-otel-collector

Windows

Start powershell as administrator and run the following command:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\observiq-otel-collector" /v Environment /t REG_MULTI_SZ /d "GOOGLE_APPLICATION_CREDENTIALS=<path to key file>" /f

Then restart the collector service:

Restart-Service observiq-otel-collector

MacOS

Add GOOGLE_APPLICATION_CREDENTIALS to the EnvironmentVariables dict in the launchd service file /Library/LaunchDaemons/com.observiq.collector.plist (other values are shown for context):

<key>EnvironmentVariables</key>
<dict>
    <key>GOOGLE_APPLICATION_CREDENTIALS</key>
    <string><path to key file></string>
    <key>OIQ_OTEL_COLLECTOR_HOME</key>
    <string>/opt/observiq-otel-collector/</string>
    <key>OIQ_OTEL_COLLECTOR_STORAGE</key>
    <string>/opt/observiq-otel-collector/storage</string>
</dict>

Then restart the collector:

sudo launchctl unload /Library/LaunchDaemons/com.observiq.collector.plist
sudo launchctl load /Library/LaunchDaemons/com.observiq.collector.plist

Example Configuration

In this example, we are parsing log data from the given PubSub subscription.

Web Interface

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
    name: pubsubpush
spec:
    type: googlecloudpubsubpush:1
    parameters:
        - name: listen_address
          value: 0.0.0.0
        - name: listen_port
          value: 8090
        - name: encoding_extension
          value:
            disabled: false
            parameters:
                - name: cloud_logging_json_handling
                  value: json
                - name: cloud_logging_proto_handling
                  value: json
            type: googlecloudlogentry_encoding
        - name: enable_tls
          value: false
        - name: cert_file
          value: ""
        - name: key_file
          value: ""
        - name: mutual_tls
          value: false
        - name: ca_file
          value: ""
        - name: include_metadata
          value: false
        - name: enable_cors
          value: false
        - name: cors_allowed_origins
          value: []
        - name: cors_allowed_headers
          value: []
        - name: cors_max_age
          value: 0