> For the complete documentation index, see [llms.txt](https://docs.bindplane.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bindplane.com/integrations/sources/generic-rest-api.md). # Generic REST API The REST API source is a generic receiver that pulls JSON data from any REST API endpoint. It supports both logs and metrics, with configurable authentication, pagination, time-based offset tracking, and checkpointing so collection can resume after a restart. **Note:** Because of the wide range of possible use cases for this source, it offers a best-effort integration with common API patterns. It may not align with every REST API. ### Supported Telemetry Types | Platform | Metrics | Logs | Traces | | -------- | ------- | ---- | ------ | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | Use the **Choose Telemetry Type** selector to collect Logs, Metrics, or both. Metrics extraction is driven by the [Metrics](#metrics) field mapping below. ### Prerequisites * A REST API endpoint that returns JSON (a standard JSON array/object or newline-delimited JSON). * Any authentication credentials the endpoint requires (API key, bearer token, basic credentials, OAuth 2.0 client credentials, or Akamai EdgeGrid). * For metrics collection, the response must include a field that holds each metric's name. ### Configuration
Bindplane docs - Generic REST API - image 1
The UI groups parameters into tabs: **Connection**, **Metrics**, **Authentication**, **Pagination**, and **Advanced**. The tables below follow that grouping. #### Connection | Parameter | Type | Default | Description | | ------------------------- | ------------------ | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Choose Telemetry Type | Telemetry Selector | `[Logs, Metrics]` | Telemetry types to collect. Valid values: `Logs`, `Metrics`. | | URL | String | | The base URL for the REST API endpoint. Required. | | Response Format | Enum | `JSON` | Format of the API response body. Valid values: `JSON`, `NDJSON`. In NDJSON mode each line is a separate JSON object. | | Response Field | String | | Name of the field in the response that contains the array of items. If empty, the response is assumed to be a top-level array. Use dot notation for nested fields (e.g. `response.data`). Only applies when `response_format` is `JSON`. | | Start Time Parameter Name | String | | Query parameter name for the start time (e.g. `since`, `from`, `start_time`). Required when `pagination_mode` is `timestamp`. | | Start Time Value | Timestamp | | Timestamp to start collecting data from. Conditional on `start_time_param_name` being set. For timestamp pagination, this is the initial start value. If not set, collection starts from the beginning. | | End Time Parameter Name | String | | Query parameter name for the end time (e.g. `until`, `to`, `end_time`). When set, an end time is sent on every request. | | End Time Value | Timestamp | | Timestamp to stop collecting data at. Conditional on `end_time_param_name` being set. | | Timestamp Format | Enum | `RFC3339` | Format for the start and end time query parameters. Accepts Go time layout strings or epoch formats (the field is creatable, so a custom layout can be entered). See [Time-Bounding](#time-bounding). | #### Metrics These fields apply only when `Metrics` is selected in `telemetry_types`. Each value names the field in a response item that supplies the corresponding metric attribute. See [Metrics field mapping](#metrics-field-mapping) for worked examples. | Parameter | Type | Default | Description | | ----------------------------- | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Name Field | String | | Field name in each response item containing the metric name. Required when collecting Metrics. If the field is not found, the metric is dropped and a warning is logged. | | Description Field | String | | Field name containing the metric description. If unset or not found, defaults to `Metric from REST API`. | | Type Field | String | | Field name containing the metric type (`gauge`, `sum`, `histogram`, `summary`). If unset or not found, defaults to `gauge`. | | Unit Field | String | | Field name containing the metric unit. If unset or not found, no unit is set. | | Monotonic Field | String | | Field name (boolean) indicating whether a `sum` metric is monotonic. Only applies to `sum` metrics. Defaults to `false` if unset or not found. | | Aggregation Temporality Field | String | | Field name containing the aggregation temporality (`cumulative` or `delta`). Defaults to `cumulative` if unset or not found. | When a field name is configured, that field is automatically excluded from the metric's attributes to avoid duplication. #### Authentication | Parameter | Type | Default | Description | | ------------------- | ---- | --------- | -------------------------------------------------------------------------------------------------------------- | | Authentication Mode | Enum | `API Key` | Authentication mode. Valid values: `None`, `API Key`, `Bearer Token`, `Basic`, `Akamai EdgeGrid`, `OAuth 2.0`. | Each mode exposes its own credential fields, shown only when that mode is selected. **API Key (`auth_mode: API Key`)** | Parameter | Type | Default | Description | | ------------------- | ------ | ------- | ----------------------------------------------------- | | API Key Header Name | String | | Header name for API key authentication. Required. | | API Key Value | String | | API key value. Required. Stored as a sensitive value. | **Bearer Token (`auth_mode: Bearer Token`)** | Parameter | Type | Default | Description | | ------------ | ------ | ------- | ---------------------------------------------------------- | | Bearer Token | String | | Bearer token value. Required. Stored as a sensitive value. | **Basic (`auth_mode: Basic`)** | Parameter | Type | Default | Description | | --------- | ------ | ------- | ------------------------------------------------------------------------- | | Username | String | | Username for basic authentication. Required. | | Password | String | | Password for basic authentication. Required. Stored as a sensitive value. | **OAuth 2.0 (`auth_mode: OAuth 2.0`)** | Parameter | Type | Default | Description | | --------------- | ------- | ------- | -------------------------------------------------------------------------------------------------- | | Client ID | String | | Client identifier assigned during application registration. Required. Stored as a sensitive value. | | Client Secret | String | | Client secret issued during application registration. Required. Stored as a sensitive value. | | Token URL | String | | Endpoint URL used to retrieve the OAuth 2.0 access token. Required. | | Scopes | Strings | `[]` | Scope(s) of the access request. | | Endpoint Params | Map | `{}` | Additional parameters to send to the token endpoint. | **Akamai EdgeGrid (`auth_mode: Akamai EdgeGrid`)** The Akamai API requires an enterprise license. This authentication method has not been tested against an Akamai API. | Parameter | Type | Default | Description | | ------------------ | ------ | ------- | ------------------------------------------------------------------- | | Access Token | String | | EdgeGrid access token. Required. Stored as a sensitive value. | | Client Token | String | | EdgeGrid client token. Required. Stored as a sensitive value. | | Client Secret | String | | EdgeGrid client secret. Required. Stored as a sensitive value. | | Account Switch Key | String | | Account switch key used to make requests against a managed account. | #### Pagination | Parameter | Type | Default | Description | | -------------------------- | ------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Pagination Mode | Enum | `none` | Pagination mode. Valid values: `none`, `offset`, `page`, `timestamp`. | | Pagination Response Source | Enum | `Body` | Where to read pagination metadata from. Valid values: `Body`, `Header`. Use `Header` when the API returns pagination metadata in HTTP response headers. Conditional on `pagination_mode` not being `none`. | | Total Record Count Field | String | | Name of the field or header containing the total record count. Conditional on `pagination_mode` not being `none`. | | Page Limit | Integer | `0` | Maximum number of pages to fetch (`0` = no limit). Conditional on `pagination_mode` not being `none`. | | Zero-Based Index | Boolean | `false` | Indicates that the requested data starts at index 0. Conditional on `pagination_mode` not being `none`. | **Offset / Limit (`pagination_mode: offset`)** | Parameter | Type | Default | Description | | ---------------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Offset Field Name | String | `offset` | Query parameter name for offset. | | Limit Field Name | String | | Query parameter name for limit. | | Starting Offset | Integer | `0` | Starting offset value. | | Next Offset Field Name | String | | Response body field containing the next offset token. When set, the receiver uses token-based (cursor) pagination instead of numeric offsets. Supports nested fields with dot notation (e.g. `pagination.next_cursor`). | **Page / Size (`pagination_mode: page`)** | Parameter | Type | Default | Description | | ---------------------- | ------- | ------- | ----------------------------------------------------------- | | Page Number Field Name | String | `page` | Query parameter name for page number. | | Page Size Field Name | String | | Query parameter name for page size. | | Starting Page | Integer | `1` | Starting page number. | | Total Pages Field Name | String | | Field name in the response containing the total page count. | **Timestamp (`pagination_mode: timestamp`)** Timestamp pagination sends an advancing start time using the top-level `start_time_param_name` and `timestamp_format` fields. The start time advances automatically from response data as pages are consumed, so set `start_time_param_name` (Connection tab) when using this mode. | Parameter | Type | Default | Description | | -------------------- | ------- | ------- | --------------------------------------------------------------------------------------------- | | Timestamp Field Name | String | | Field name in each response item containing the timestamp (e.g. `ts`, `timestamp`). Required. | | Page Size Field Name | String | | Query parameter name for page size (e.g. `perPage`, `limit`). | | Page Size | Integer | `100` | Page size to use. | #### Advanced | Parameter | Type | Default | Description | | ------------------ | ------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | Min Poll Interval | Integer | `10` | Minimum interval between API polls, in seconds. The receiver resets to this interval when data is received. Increase it to avoid hitting API rate limits. | | Max Poll Interval | Integer | `300` | Maximum interval between API polls, in seconds. The receiver uses adaptive polling that backs off when no data is returned, up to this maximum. | | Backoff Multiplier | String | `2.0` | Multiplier for increasing the poll interval when no data or a partial page is returned. Must be greater than 1.0. | | Enable Checkpoint | Boolean | `true` | Enable checkpointing so the receiver resumes from where it left off after a restart. See [Checkpointing](#checkpointing). | | Storage Directory | String | `${OIQ_OTEL_COLLECTOR_HOME}/storage` | Directory for storing checkpoint state. Conditional on `enable_storage` being `true`. | | Timeout | Integer | `10` | HTTP client timeout, in seconds. | | Headers | Map | `{}` | Custom headers to include with each request. Use the sensitive toggle for headers containing secrets. | ### Example Configuration #### Standalone Source ```yaml apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: rest_api name: rest_api spec: type: rest_api parameters: - name: telemetry_types value: - Logs - name: url value: 'https://api.example.com/v1/events' - name: response_format value: 'JSON' - name: response_field value: 'data' - name: auth_mode value: 'API Key' - name: apikey_header_name value: 'Authorization' - name: apikey_value value: 'my-api-key' - name: pagination_mode value: 'offset' - name: pagination_offset_limit_offset_field_name value: 'offset' - name: pagination_offset_limit_limit_field_name value: 'limit' - name: min_poll_interval value: '30' - name: max_poll_interval value: '300' - name: enable_storage value: 'true' ``` ### Configuration Tips #### Mapping the response body * If the API returns a top-level JSON array, leave `response_field` empty. * If the records are wrapped in an object (for example `{ "data": [ ... ], "total": 2 }`), set `response_field` to the field that holds the array, using dot notation for nested fields (e.g. `result.items`). * For newline-delimited responses, set `response_format` to `NDJSON`. Each line is treated as a separate JSON object and `response_field` is ignored. #### Choosing a pagination mode * `offset` suits APIs that page with numeric `offset`/`limit` query parameters, or cursor APIs when you set `pagination_offset_limit_next_offset_field_name`. * `page` suits APIs that page with a page number and page size. * `timestamp` suits APIs that page by an advancing time window. It requires `start_time_param_name` (Connection tab) and `pagination_timestamp_timestamp_field_name`. * When pagination metadata is returned in HTTP headers rather than the body, set `pagination_response_source` to `Header`. #### Polling and rate limits * The receiver polls adaptively between `min_poll_interval` and `max_poll_interval`, backing off by `backoff_multiplier` when a poll returns no data. * Raise `min_poll_interval` if you are hitting the API's rate limit. Lower `max_poll_interval` if you need fresher data when the endpoint is idle. ### Time-Bounding The Connection-tab time fields add start and/or end time query parameters to every request. They work with any pagination mode (or none). When used with `timestamp` pagination, the start time advances automatically through response data. `start_time_value` accepts `now` (resolved once at receiver start, not dynamically updated) or a fixed timestamp in the configured format. `timestamp_format` controls how both start and end times are rendered. Supported formats: * `RFC3339` (`2006-01-02T15:04:05Z07:00`) - default * `ISO8601 Basic` (`20060102T150405`) * `24-hour datetime` (`2006-01-02 15:04:05`) * `Epoch Seconds`, `Epoch Milliseconds`, `Epoch Microseconds`, `Epoch Nanoseconds` - numeric epoch values The format field is creatable, so a custom Go time layout can be entered if none of the presets match the API. ### Response Format The receiver expects JSON responses in one of two shapes. Top-level array (leave `response_field` empty): ```json [ { "id": "1", "message": "log entry 1" }, { "id": "2", "message": "log entry 2" } ] ``` Object with a data field (set `response_field: data`): ```json { "data": [ { "id": "1", "message": "log entry 1" }, { "id": "2", "message": "log entry 2" } ], "total": 2 } ``` ### Metrics field mapping When `Metrics` is selected, the receiver builds one metric per response item using the field names you map. Given this response: ```json { "data": [ { "name": "queue.depth", "value": 42, "unit": "{items}", "kind": "gauge", "help": "Current queue depth" } ] } ``` set the following so each item becomes a metric: * `response_field`: `data` * `metrics_name_field`: `name` * `metrics_type_field`: `kind` * `metrics_unit_field`: `unit` * `metrics_description_field`: `help` The metric is emitted as `queue.depth` (gauge, unit `{items}`). The mapped fields (`name`, `kind`, `unit`, `help`) are excluded from the metric's attributes; remaining fields such as `value` supply the data point. If `metrics_name_field` is not present in an item, that item is dropped and a warning is logged. ### Checkpointing When checkpointing is enabled (`enable_storage: true`, on by default), the receiver writes its pagination state to a file storage extension at `storage_directory`. After a restart it resumes from where it left off, preventing duplicate collection. The checkpoint includes the current pagination state (offset, page number, or timestamp) and the number of pages fetched. For timestamp pagination, the advancing start time is persisted so the receiver continues from the most recently observed timestamp. Each checkpoint is tagged with a fingerprint derived from the config fields that define what data the receiver fetches: `url`, pagination mode, `start_time_value`, `end_time_value`, the offset starting value, and the page starting value. If any of these change between runs, the stored checkpoint is discarded because it tracks state for a different query. Other changes (credentials, polling intervals) do not invalidate the checkpoint. ### Troubleshooting **Symptom:** No data is collected and the collector logs show authentication failures (401/403). **Solution:** Confirm `auth_mode` matches what the API expects and that the credential fields are correct. For API Key auth, verify `apikey_header_name` is the exact header the API requires (for example `Authorization` or `X-API-Key`). **Symptom:** The endpoint returns data but no records are produced. **Solution:** Check `response_field`. If the records are nested inside an object, set `response_field` to the field (or dotted path) that holds the array. For newline-delimited responses, set `response_format` to `NDJSON` and leave `response_field` empty. **Symptom:** Metrics are dropped with a warning in the collector logs. **Solution:** The `metrics_name_field` value must name a field present in every response item. Set it to the field that holds the metric name, and confirm `metrics_type_field` (if set) resolves to `gauge`, `sum`, `histogram`, or `summary`. **Symptom:** The same records are collected repeatedly after a restart, or pagination never advances. **Solution:** Ensure `enable_storage` is on so checkpoints persist. If you intentionally changed the query (URL, pagination mode, or start/end time), the prior checkpoint is discarded by design. For timestamp pagination, verify `pagination_timestamp_timestamp_field_name` names a field present in each item so the start time can advance. **Symptom:** The API returns HTTP 429 (rate limited). **Solution:** Raise `min_poll_interval` so the receiver polls less aggressively, and lower `pagination_page_limit` or the page size to fetch fewer records per cycle. ### Related Resources * [Bindplane source documentation](https://bindplane.com/docs/resources/sources/generic-rest-api) * [Storage extension and checkpointing](https://docs.bindplane.com/docs/resources/extensions) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.bindplane.com/integrations/sources/generic-rest-api.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.