# Generic REST API The REST API source is a generic receiver that can pull JSON data from any REST API endpoint. It supports both logs and metrics collection, with configurable authentication, pagination, and time-based offset tracking. **Note**: Due to the wide range of possible use cases for this source, it offers a best-effort integration with common API patterns, but may not completely align with every REST API. ## Supported Platforms | Platform | Metrics | Logs | Traces | | -------- | ------- | ---- | ------ | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites * A REST API endpoint that returns JSON data * Appropriate authentication credentials (if required) ## Configuration Table | Field | Type | Default | Required | Description | | -------------------- | --------- | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | string | | `true` | The base URL for the REST API endpoint. | | `response_format` | string | `json` | `false` | Response body format: `json` (standard JSON array/object) or `ndjson` (newline-delimited JSON). In NDJSON mode, each line is a separate JSON object. | | `response_field` | string | | `false` | The name of the field in the response that contains the array of items. If empty, the response is assumed to be a top-level array. For nested fields, use dot notation (e.g., `response.data`). Not used when `response_format` is `ndjson`. | | `metrics` | object | | `false` | [Metrics configuration](#metrics-configuration) | | `auth_mode` | string | `none` | `false` | Authentication mode: `none`, `apikey`, `bearer`, `basic`, `oauth2`, or `akamai_edgegrid`. | | `apikey` | object | | `false` | [API Key configuration](#api-key) | | `bearer` | object | | `false` | [Bearer Token configuration](#bearer-token) | | `basic` | object | | `false` | [Basic Auth configuration](#basic-auth) | | `oauth2` | object | | `false` | [OAuth2.0 Client Credentials configuration](#oauth-2.0-client-credentials) | | `akamai_edgegrid` | object | | `false` | [Akamai EdgeGrid configuration](#akamai-edgegrid) | | `headers` | map | | `false` | A map of custom headers to send with each request. Header values will appear in plaintext unless marked as `Sensitive`. | | `pagination` | object | | `false` | [Pagination configuration](#pagination-configuration) | | Time-bounding fields | — | | `false` | Top-level `start_time_param_name`, `start_time_value`, `end_time_param_name`, `end_time_value`, and `timestamp_format` fields. See [Time-Bounding Configuration](#time-bounding-configuration). | | `min_poll_interval` | duration | `10s` | `false` | Minimum interval between API polls. The receiver resets to this interval when data is received. Increase this to prevent hitting API rate limits. | | `max_poll_interval` | duration | `5m` | `false` | Maximum interval between API polls. The receiver uses adaptive polling that starts at `min_poll_interval` and backs off when no data is returned, up to this maximum. | | `backoff_multiplier` | float | `2.0` | `false` | Multiplier for increasing the poll interval when no data or a partial page is returned. Must be greater than 1.0. | | `storage` | component | | `false` | The component ID of a storage extension for checkpointing. | | `timeout` | duration | `10s` | `false` | HTTP client timeout. | ### Auth Mode Configuration #### **None (No Authentication)** Use `auth_mode: none` for public APIs that don't require authentication. No additional configuration is needed. #### **API Key** | Field | Type | Default | Required | Description | | ------------- | ------ | ------- | -------- | ------------------------ | | `header_name` | string | | `true` | Header name for API key. | | `value` | string | | `true` | API key value. | #### **Bearer Token** | Field | Type | Default | Required | Description | | ------- | ------ | ------- | -------- | ------------------- | | `token` | string | | `true` | Bearer token value. | #### **Basic Auth** | Field | Type | Default | Required | Description | | ---------- | ------ | ------- | -------- | ------------------------ | | `username` | string | | `true` | Username for basic auth. | | `password` | string | | `true` | Password for basic auth. | #### **OAuth 2.0 Client Credentials** | Field | Type | Default | Required | Description | | ----------------- | ------------------ | ------- | -------- | ---------------------------------------------------- | | `client_id` | string | | `true` | OAuth2.0 client ID. | | `client_secret` | string | | `true` | OAuth2.0 client secret. | | `token_url` | string | | `true` | OAuth2.0 token endpoint URL. | | `scopes` | \[]string | | `false` | OAuth2.0 scopes to request. | | `endpoint_params` | map\[string]string | | `false` | Additional parameters to send to the token endpoint. | #### **Akamai EdgeGrid** **The Akamai API requires an enterprise license. This authentication method has not been tested against an Akamai API.** | Field | Type | Default | Required | Description | | ---------------------- | ------ | ------- | -------- | ------------------------------ | | `akamai_access_token` | string | | `true` | Akamai EdgeGrid access token. | | `akamai_client_token` | string | | `true` | Akamai EdgeGrid client token. | | `akamai_client_secret` | string | | `true` | Akamai EdgeGrid client secret. | ### Time-Bounding Configuration These top-level fields add start and/or end time query parameters to every request. They work with **any** pagination mode (or none). When used with `timestamp` pagination, the start time advances automatically through response data. | Field | Type | Default | Required | Description | | ----------------------- | ------ | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `start_time_param_name` | string | | `false` | Query parameter name for start time (e.g., "since", "from", "start\_time"). **Required** when `pagination.mode` is `timestamp`. | | `start_time_value` | string | | `false` | Value for the start time: `"now"` resolves to the current time at receiver start (not dynamically updated), or a fixed timestamp in the configured format (e.g., `"2025-01-01T00:00:00Z"` or `"1704067200"` for epoch). For timestamp pagination, this is the initial start value. | | `end_time_param_name` | string | | `false` | Query parameter name for end time (e.g., "until", "to", "end\_time"). If set, sends an end time on every request. | | `end_time_value` | string | `now` | `false` | Value for the end time: `"now"` (default) resolves to the current time at receiver start (not dynamically updated), or a fixed timestamp in the configured format. | | `timestamp_format` | string | RFC3339 | `false` | Format for both start and end time query parameters. Accepts Go time format strings or epoch formats (see [Timestamp Formats](#timestamp-based-pagination)). | ### Pagination Configuration | Field | Type | Default | Required | Description | | ------------------------------------- | ------ | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pagination.mode` | string | `none` | `false` | Pagination mode: `none`, `offset_limit`, `page_size`, or `timestamp`. | | `pagination.response_source` | string | `body` | `false` | Where to extract pagination response attributes from: `body` (response body or NDJSON metadata line) or `header` (HTTP response headers). Applies to `total_record_count_field`, `next_offset_field_name`, and `total_pages_field_name`. | | `pagination.total_record_count_field` | string | | `false` | Name of the field or header containing total record count. | | `pagination.page_limit` | int | `0` | `false` | Maximum number of pages to fetch (0 = no limit). | | `pagination.zero_based_index` | bool | `false` | `false` | Indicates that the requested data starts at index 0. | #### **Offset/Limit Pagination** | Field | Type | Default | Required | Description | | ------------------------------------------------ | ------ | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pagination.offset_limit.offset_field_name` | string | | `false` | Query parameter name for offset. | | `pagination.offset_limit.limit_field_name` | string | | `false` | Query parameter name for limit. | | `pagination.offset_limit.starting_offset` | int | `0` | `false` | Starting offset value. | | `pagination.offset_limit.next_offset_field_name` | string | | `false` | Name of the field or header containing the next offset token. When set, the receiver uses token-based (cursor) pagination instead of numeric offsets. For body sources, supports nested fields with dot notation (e.g., `pagination.next_cursor`). | #### **Page/Size Pagination** | Field | Type | Default | Required | Description | | --------------------------------------------- | ------ | ------- | -------- | -------------------------------------------------------- | | `pagination.page_size.page_num_field_name` | string | | `false` | Query parameter name for page number. | | `pagination.page_size.page_size_field_name` | string | | `false` | Query parameter name for page size. | | `pagination.page_size.starting_page` | int | `1` | `false` | Starting page number. | | `pagination.page_size.total_pages_field_name` | string | | `false` | Name of the field or header containing total page count. | #### **Timestamp-Based Pagination** Timestamp pagination uses the top-level `start_time_param_name` and `timestamp_format` fields to send an advancing start time. The start time is automatically updated from response data as pages are consumed. | Field | Type | Default | Required | Description | | ------------------------------------------- | ------ | ------- | -------- | ------------------------------------------------------------------------------------ | | `pagination.timestamp.timestamp_field_name` | string | | `true` | Field name in each response item containing the timestamp (e.g., "ts", "timestamp"). | | `pagination.timestamp.page_size_field_name` | string | | `false` | Query parameter name for page size (e.g., "perPage", "limit"). | | `pagination.timestamp.page_size` | int | `100` | `false` | Page size to use. | Common timestamp formats: * `2006-01-02T15:04:05Z07:00` - RFC3339 (default) * `20060102150405` - YYYYMMDDHHMMSS * `2006-01-02 15:04:05` - Date and time with space separator * `2006-01-02` - Date only Epoch timestamp formats (sends numeric values instead of formatted strings): * `epoch_s` - Unix epoch seconds (e.g., `1704067200`) * `epoch_ms` - Unix epoch milliseconds (e.g., `1704067200000`) * `epoch_us` - Unix epoch microseconds (e.g., `1704067200000000`) * `epoch_ns` - Unix epoch nanoseconds (e.g., `1704067200000000000`) * `epoch_s_frac` - Fractional epoch seconds (e.g., `1704067200.123456`). The integer part is seconds and the fractional digits represent sub-second precision (`.123` = milliseconds, `.123456` = microseconds, `.123456789` = nanoseconds). Used by APIs that expect `seconds.fraction` format in both responses and query parameters. ### Metrics Configuration The metrics configuration allows you to customize how metrics are extracted from API responses. | Field | Type | Default | Required | Description | | --------------------------------------- | ------ | ------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metrics.name_field` | string | | `true` | Field name in each response item containing the metric name. If not found, the metric will be dropped and a warning will be logged. | | `metrics.description_field` | string | | `false` | Field name in each response item containing the metric description. If not specified or not found, defaults to `Metric from REST API`. | | `metrics.type_field` | string | | `false` | Field name in each response item containing the metric type (`gauge`, `sum`, `histogram`, `summary`). If not specified or not found, defaults to `gauge`. | | `metrics.unit_field` | string | | `false` | Field name in each response item containing the metric unit. If not specified or not found, no unit is set. | | `metrics.monotonic_field` | string | | `false` | Field name in each response item indicating if a sum metric is monotonic (boolean). Only applies to `sum` metrics. If not specified or not found, defaults to `false` for safety. | | `metrics.aggregation_temporality_field` | string | | `false` | Field name in each response item containing the aggregation temporality (`cumulative` or `delta`). If not specified or not found, defaults to `cumulative`. | **Note:** When field names are configured, those fields are automatically excluded from metric attributes to avoid duplication. If the `name_field` isn't found, the metric will be dropped. ### Response Format The receiver expects JSON responses in one of two formats: 1. **Top-level array:** ```json [ { "id": "1", "message": "log entry 1" }, { "id": "2", "message": "log entry 2" } ] ``` 2. **Object with data field:** ```json { "data": [ { "id": "1", "message": "log entry 1" }, { "id": "2", "message": "log entry 2" } ], "total": 2 } ``` When using the second format, specify the field name in `response_field` (e.g., `"data"`). ### Checkpointing When a storage extension is configured via the `storage` field, the receiver saves its pagination state to storage. This allows the receiver to resume from where it left off after a restart, preventing duplicate data collection. The checkpoint includes: * Current pagination state (offset/page number/timestamp) * Number of pages fetched For timestamp-based pagination, the advancing start time is persisted so the receiver continues from the most recently observed timestamp after a restart. #### Config Fingerprint Each checkpoint is tagged with a fingerprint derived from the config fields that define what data the receiver fetches: `url`, `pagination.mode`, `start_time_value`, `end_time_value`, `pagination.offset_limit.starting_offset`, and `pagination.page_size.starting_page`. If any of these change between runs, the stored checkpoint is considered invalid and discarded, because it tracks pagination state for a different query. Other config changes (such as credentials or polling intervals) do not invalidate the checkpoint. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.bindplane.com/integrations/sources/generic-rest-api.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.