Generic REST API

The REST API source is a generic receiver that pulls JSON data from any REST API endpoint. It supports both logs and metrics, with configurable authentication, pagination, time-based offset tracking, and checkpointing so collection can resume after a restart.

Note: Because of the wide range of possible use cases for this source, it offers a best-effort integration with common API patterns. It may not align with every REST API.

Supported Telemetry Types

Platform	Metrics	Logs	Traces
Linux	✓	✓
Windows	✓	✓
macOS	✓	✓

Use the Choose Telemetry Type selector to collect Logs, Metrics, or both. Metrics extraction is driven by the Metrics field mapping below.

Prerequisites

• A REST API endpoint that returns JSON (a standard JSON array/object or newline-delimited JSON).
• Any authentication credentials the endpoint requires (API key, bearer token, basic credentials, OAuth 2.0 client credentials, or Akamai EdgeGrid).
• For metrics collection, the response must include a field that holds each metric's name.

Configuration

The UI groups parameters into tabs: Connection, Metrics, Authentication, Pagination, and Advanced. The tables below follow that grouping.

Connection

Parameter	Type	Default	Description
Choose Telemetry Type	Telemetry Selector	[Logs, Metrics]	Telemetry types to collect. Valid values: Logs, Metrics.
URL	String		The base URL for the REST API endpoint. Required.
Response Format	Enum	JSON	Format of the API response body. Valid values: JSON, NDJSON. In NDJSON mode each line is a separate JSON object.
Response Field	String		Name of the field in the response that contains the array of items. If empty, the response is assumed to be a top-level array. Use dot notation for nested fields (e.g. response.data). Only applies when response_format is JSON.
Start Time Parameter Name	String		Query parameter name for the start time (e.g. since, from, start_time). Required when pagination_mode is timestamp.
Start Time Value	Timestamp		Timestamp to start collecting data from. Conditional on start_time_param_name being set. For timestamp pagination, this is the initial start value. If not set, collection starts from the beginning.
End Time Parameter Name	String		Query parameter name for the end time (e.g. until, to, end_time). When set, an end time is sent on every request.
End Time Value	Timestamp		Timestamp to stop collecting data at. Conditional on end_time_param_name being set.
Timestamp Format	Enum	RFC3339	Format for the start and end time query parameters. Accepts Go time layout strings or epoch formats (the field is creatable, so a custom layout can be entered). See Time-Bounding.

Metrics

These fields apply only when Metrics is selected in telemetry_types. Each value names the field in a response item that supplies the corresponding metric attribute. See Metrics field mapping for worked examples.

Parameter	Type	Default	Description
Name Field	String		Field name in each response item containing the metric name. Required when collecting Metrics. If the field is not found, the metric is dropped and a warning is logged.
Description Field	String		Field name containing the metric description. If unset or not found, defaults to Metric from REST API.
Type Field	String		Field name containing the metric type (gauge, sum, histogram, summary). If unset or not found, defaults to gauge.
Unit Field	String		Field name containing the metric unit. If unset or not found, no unit is set.
Monotonic Field	String		Field name (boolean) indicating whether a sum metric is monotonic. Only applies to sum metrics. Defaults to false if unset or not found.
Aggregation Temporality Field	String		Field name containing the aggregation temporality (cumulative or delta). Defaults to cumulative if unset or not found.

When a field name is configured, that field is automatically excluded from the metric's attributes to avoid duplication.

Authentication

Parameter	Type	Default	Description
Authentication Mode	Enum	API Key	Authentication mode. Valid values: None, API Key, Bearer Token, Basic, Akamai EdgeGrid, OAuth 2.0.

Each mode exposes its own credential fields, shown only when that mode is selected.

API Key (auth_mode: API Key)

Parameter	Type	Default	Description
API Key Header Name	String		Header name for API key authentication. Required.
API Key Value	String		API key value. Required. Stored as a sensitive value.

Bearer Token (auth_mode: Bearer Token)

Parameter	Type	Default	Description
Bearer Token	String		Bearer token value. Required. Stored as a sensitive value.

Basic (auth_mode: Basic)

Parameter	Type	Default	Description
Username	String		Username for basic authentication. Required.
Password	String		Password for basic authentication. Required. Stored as a sensitive value.

OAuth 2.0 (auth_mode: OAuth 2.0)

Parameter	Type	Default	Description
Client ID	String		Client identifier assigned during application registration. Required. Stored as a sensitive value.
Client Secret	String		Client secret issued during application registration. Required. Stored as a sensitive value.
Token URL	String		Endpoint URL used to retrieve the OAuth 2.0 access token. Required.
Scopes	Strings	[]	Scope(s) of the access request.
Endpoint Params	Map	{}	Additional parameters to send to the token endpoint.

Akamai EdgeGrid (auth_mode: Akamai EdgeGrid)

The Akamai API requires an enterprise license. This authentication method has not been tested against an Akamai API.

Parameter	Type	Default	Description
Access Token	String		EdgeGrid access token. Required. Stored as a sensitive value.
Client Token	String		EdgeGrid client token. Required. Stored as a sensitive value.
Client Secret	String		EdgeGrid client secret. Required. Stored as a sensitive value.
Account Switch Key	String		Account switch key used to make requests against a managed account.

Pagination

Parameter	Type	Default	Description
Pagination Mode	Enum	none	Pagination mode. Valid values: none, offset, page, timestamp.
Pagination Response Source	Enum	Body	Where to read pagination metadata from. Valid values: Body, Header. Use Header when the API returns pagination metadata in HTTP response headers. Conditional on pagination_mode not being none.
Total Record Count Field	String		Name of the field or header containing the total record count. Conditional on pagination_mode not being none.
Page Limit	Integer	0	Maximum number of pages to fetch (0 = no limit). Conditional on pagination_mode not being none.
Zero-Based Index	Boolean	false	Indicates that the requested data starts at index 0. Conditional on pagination_mode not being none.

Offset / Limit (pagination_mode: offset)

Parameter	Type	Default	Description
Offset Field Name	String	offset	Query parameter name for offset.
Limit Field Name	String		Query parameter name for limit.
Starting Offset	Integer	0	Starting offset value.
Next Offset Field Name	String		Response body field containing the next offset token. When set, the receiver uses token-based (cursor) pagination instead of numeric offsets. Supports nested fields with dot notation (e.g. pagination.next_cursor).

Page / Size (pagination_mode: page)

Parameter	Type	Default	Description
Page Number Field Name	String	page	Query parameter name for page number.
Page Size Field Name	String		Query parameter name for page size.
Starting Page	Integer	1	Starting page number.
Total Pages Field Name	String		Field name in the response containing the total page count.

Timestamp (pagination_mode: timestamp)

Timestamp pagination sends an advancing start time using the top-level start_time_param_name and timestamp_format fields. The start time advances automatically from response data as pages are consumed, so set start_time_param_name (Connection tab) when using this mode.

Parameter	Type	Default	Description
Timestamp Field Name	String		Field name in each response item containing the timestamp (e.g. ts, timestamp). Required.
Page Size Field Name	String		Query parameter name for page size (e.g. perPage, limit).
Page Size	Integer	100	Page size to use.

Advanced

Parameter	Type	Default	Description
Min Poll Interval	Integer	10	Minimum interval between API polls, in seconds. The receiver resets to this interval when data is received. Increase it to avoid hitting API rate limits.
Max Poll Interval	Integer	300	Maximum interval between API polls, in seconds. The receiver uses adaptive polling that backs off when no data is returned, up to this maximum.
Backoff Multiplier	String	2.0	Multiplier for increasing the poll interval when no data or a partial page is returned. Must be greater than 1.0.
Enable Checkpoint	Boolean	true	Enable checkpointing so the receiver resumes from where it left off after a restart. See Checkpointing.
Storage Directory	String	${OIQ_OTEL_COLLECTOR_HOME}/storage	Directory for storing checkpoint state. Conditional on enable_storage being true.
Timeout	Integer	10	HTTP client timeout, in seconds.
Headers	Map	{}	Custom headers to include with each request. Use the sensitive toggle for headers containing secrets.

Example Configuration

Standalone Source

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: rest_api
  name: rest_api
spec:
  type: rest_api
  parameters:
    - name: telemetry_types
      value:
        - Logs
    - name: url
      value: 'https://api.example.com/v1/events'
    - name: response_format
      value: 'JSON'
    - name: response_field
      value: 'data'
    - name: auth_mode
      value: 'API Key'
    - name: apikey_header_name
      value: 'Authorization'
    - name: apikey_value
      value: 'my-api-key'
    - name: pagination_mode
      value: 'offset'
    - name: pagination_offset_limit_offset_field_name
      value: 'offset'
    - name: pagination_offset_limit_limit_field_name
      value: 'limit'
    - name: min_poll_interval
      value: '30'
    - name: max_poll_interval
      value: '300'
    - name: enable_storage
      value: 'true'

Configuration Tips

Mapping the response body

• If the API returns a top-level JSON array, leave response_field empty.
• If the records are wrapped in an object (for example { "data": [ ... ], "total": 2 }), set response_field to the field that holds the array, using dot notation for nested fields (e.g. result.items).
• For newline-delimited responses, set response_format to NDJSON. Each line is treated as a separate JSON object and response_field is ignored.

Choosing a pagination mode

• offset suits APIs that page with numeric offset/limit query parameters, or cursor APIs when you set pagination_offset_limit_next_offset_field_name.
• page suits APIs that page with a page number and page size.
• timestamp suits APIs that page by an advancing time window. It requires start_time_param_name (Connection tab) and pagination_timestamp_timestamp_field_name.
• When pagination metadata is returned in HTTP headers rather than the body, set pagination_response_source to Header.

Polling and rate limits

• The receiver polls adaptively between min_poll_interval and max_poll_interval, backing off by backoff_multiplier when a poll returns no data.
• Raise min_poll_interval if you are hitting the API's rate limit. Lower max_poll_interval if you need fresher data when the endpoint is idle.

Time-Bounding

The Connection-tab time fields add start and/or end time query parameters to every request. They work with any pagination mode (or none). When used with timestamp pagination, the start time advances automatically through response data.

start_time_value accepts now (resolved once at receiver start, not dynamically updated) or a fixed timestamp in the configured format. timestamp_format controls how both start and end times are rendered. Supported formats:

• RFC3339 (2006-01-02T15:04:05Z07:00) - default
• ISO8601 Basic (20060102T150405)
• 24-hour datetime (2006-01-02 15:04:05)
• Epoch Seconds, Epoch Milliseconds, Epoch Microseconds, Epoch Nanoseconds - numeric epoch values

The format field is creatable, so a custom Go time layout can be entered if none of the presets match the API.

Response Format

The receiver expects JSON responses in one of two shapes.

Top-level array (leave response_field empty):

[
  { "id": "1", "message": "log entry 1" },
  { "id": "2", "message": "log entry 2" }
]

Object with a data field (set response_field: data):

{
  "data": [
    { "id": "1", "message": "log entry 1" },
    { "id": "2", "message": "log entry 2" }
  ],
  "total": 2
}

Metrics field mapping

When Metrics is selected, the receiver builds one metric per response item using the field names you map. Given this response:

{
  "data": [
    {
      "name": "queue.depth",
      "value": 42,
      "unit": "{items}",
      "kind": "gauge",
      "help": "Current queue depth"
    }
  ]
}

set the following so each item becomes a metric:

• response_field: data
• metrics_name_field: name
• metrics_type_field: kind
• metrics_unit_field: unit
• metrics_description_field: help

The metric is emitted as queue.depth (gauge, unit {items}). The mapped fields (name, kind, unit, help) are excluded from the metric's attributes; remaining fields such as value supply the data point. If metrics_name_field is not present in an item, that item is dropped and a warning is logged.

Checkpointing

When checkpointing is enabled (enable_storage: true, on by default), the receiver writes its pagination state to a file storage extension at storage_directory. After a restart it resumes from where it left off, preventing duplicate collection. The checkpoint includes the current pagination state (offset, page number, or timestamp) and the number of pages fetched. For timestamp pagination, the advancing start time is persisted so the receiver continues from the most recently observed timestamp.

Each checkpoint is tagged with a fingerprint derived from the config fields that define what data the receiver fetches: url, pagination mode, start_time_value, end_time_value, the offset starting value, and the page starting value. If any of these change between runs, the stored checkpoint is discarded because it tracks state for a different query. Other changes (credentials, polling intervals) do not invalidate the checkpoint.

Troubleshooting

Symptom: No data is collected and the collector logs show authentication failures (401/403). Solution: Confirm auth_mode matches what the API expects and that the credential fields are correct. For API Key auth, verify apikey_header_name is the exact header the API requires (for example Authorization or X-API-Key).

Symptom: The endpoint returns data but no records are produced. Solution: Check response_field. If the records are nested inside an object, set response_field to the field (or dotted path) that holds the array. For newline-delimited responses, set response_format to NDJSON and leave response_field empty.

Symptom: Metrics are dropped with a warning in the collector logs. Solution: The metrics_name_field value must name a field present in every response item. Set it to the field that holds the metric name, and confirm metrics_type_field (if set) resolves to gauge, sum, histogram, or summary.

Symptom: The same records are collected repeatedly after a restart, or pagination never advances. Solution: Ensure enable_storage is on so checkpoints persist. If you intentionally changed the query (URL, pagination mode, or start/end time), the prior checkpoint is discarded by design. For timestamp pagination, verify pagination_timestamp_timestamp_field_name names a field present in each item so the start time can advance.

Symptom: The API returns HTTP 429 (rate limited). Solution: Raise min_poll_interval so the receiver polls less aggressively, and lower pagination_page_limit or the page size to fetch fewer records per cycle.

Related Resources

• Storage extension and checkpointing