Generic REST API

The REST API source is a generic receiver that can pull JSON data from any REST API endpoint. It supports both logs and metrics collection, with configurable authentication, pagination, and time-based offset tracking.

Note: Due to the wide range of possible use cases for this source, it offers a best-effort integration with common API patterns, but may not completely align with every REST API.

Supported Platforms

Platform	Metrics	Logs	Traces
Linux	✓	✓
Windows	✓	✓
macOS	✓	✓

Prerequisites

• A REST API endpoint that returns JSON data

• Appropriate authentication credentials (if required)

Configuration Table

Field	Type	Default	Required	Description
url	string		true	The base URL for the REST API endpoint.
response_field	string		false	The name of the field in the response that contains the array of items. If empty, the response is assumed to be a top-level array. For nested fields, use dot notation (e.g., response.data)
metrics	object		false	Metrics configuration
auth_mode	string	none	false	Authentication mode: none, apikey, bearer, basic, oauth2, or akamai_edgegrid.
apikey	object		false	API Key configuration
bearer	object		false	Bearer Token configuration
basic	object		false	Basic Auth configuration
oauth2	object		false	OAuth2.0 Client Credentials configuration
akamai_edgegrid	object		false	Akamai EdgeGrid configuration
pagination	object		false	Pagination configuration
max_poll_interval	duration	5m	false	Maximum interval between API polls. The receiver uses adaptive polling that starts fast and backs off when no data is returned, up to this maximum.
enable_storage	bool	true	false	Configure a storage extension for checkpointing.
timeout	duration	10s	false	HTTP client timeout.

Auth Mode Configuration

None (No Authentication)

Use auth_mode: none for public APIs that don't require authentication. No additional configuration is needed.

API Key

Field	Type	Default	Required	Description
header_name	string		true	Header name for API key.
value	string		true	API key value.

Bearer Token

Field	Type	Default	Required	Description
token	string		true	Bearer token value.

Basic Auth

Field	Type	Default	Required	Description
username	string		true	Username for basic auth.
password	string		true	Password for basic auth.

OAuth 2.0 Client Credentials

Field	Type	Default	Required	Description
client_id	string		true	OAuth2.0 client ID.
client_secret	string		true	OAuth2.0 client secret.
token_url	string		true	OAuth2.0 token endpoint URL.
scopes	[]string		false	OAuth2.0 scopes to request.
endpoint_params	map[string]string		false	Additional parameters to send to the token endpoint.

Akamai EdgeGrid

The Akamai API requires an enterprise license. This authentication method has not been tested against an Akamai API.

Field	Type	Default	Required	Description
akamai_access_token	string		true	Akamai EdgeGrid access token.
akamai_client_token	string		true	Akamai EdgeGrid client token.
akamai_client_secret	string		true	Akamai EdgeGrid client secret.

Pagination Configuration

Field	Type	Default	Required	Description
pagination.mode	string	none	false	Pagination mode: none, offset_limit, page_size, or timestamp.
pagination.total_record_count_field	string		false	Field name in response containing total record count.
pagination.page_limit	int	0	false	Maximum number of pages to fetch (0 = no limit).
pagination.zero_based_index	bool	false	false	Indicates that the requested data starts at index 0.

Offset/Limit Pagination

Field	Type	Default	Required	Description
pagination.offset_limit.offset_field_name	string		false	Query parameter name for offset.
pagination.offset_limit.limit_field_name	string		false	Query parameter name for limit.
pagination.offset_limit.starting_offset	int	0	false	Starting offset value.

Page/Size Pagination

Field	Type	Default	Required	Description
pagination.page_size.page_num_field_name	string		false	Query parameter name for page number.
pagination.page_size.page_size_field_name	string		false	Query parameter name for page size.
pagination.page_size.starting_page	int	1	false	Starting page number.
pagination.page_size.total_pages_field_name	string		false	Field name in response containing total page count.

Timestamp-Based Pagination

Field	Type	Default	Required	Description
pagination.timestamp.param_name	string		true	Query parameter name for timestamp (e.g., "t0", "since", "after", "start_time").
pagination.timestamp.timestamp_field_name	string		true	Field name in each response item containing the timestamp (e.g., "ts", "timestamp").
pagination.timestamp.timestamp_format	string	RFC3339	false	Go time format string for the timestamp query parameter (e.g., "20060102150405" for YYYYMMDDHHMMSS).
pagination.timestamp.page_size_field_name	string		false	Query parameter name for page size (e.g., "perPage", "limit").
pagination.timestamp.page_size	int	100	false	Page size to use.
pagination.timestamp.initial_timestamp	string		false	Initial timestamp to start from (RFC3339 format). If not set, starts from beginning.

Common timestamp formats:

• 2006-01-02T15:04:05Z07:00 - RFC3339 (default)

• 20060102150405 - YYYYMMDDHHMMSS

• 2006-01-02 15:04:05 - Date and time with space separator

• 2006-01-02 - Date only

Metrics Configuration

The metrics configuration allows you to customize how metrics are extracted from API responses.

Field	Type	Default	Required	Description
metrics.name_field	string		true	Field name in each response item containing the metric name. If not found, the metric will be dropped and a warning will be logged.
metrics.description_field	string		false	Field name in each response item containing the metric description. If not specified or not found, defaults to Metric from REST API.
metrics.type_field	string		false	Field name in each response item containing the metric type (gauge, sum, histogram, summary). If not specified or not found, defaults to gauge.
metrics.unit_field	string		false	Field name in each response item containing the metric unit. If not specified or not found, no unit is set.
metrics.monotonic_field	string		false	Field name in each response item indicating if a sum metric is monotonic (boolean). Only applies to sum metrics. If not specified or not found, defaults to false for safety.
metrics.aggregation_temporality_field	string		false	Field name in each response item containing the aggregation temporality (cumulative or delta). If not specified or not found, defaults to cumulative.

Note: When field names are configured, those fields are automatically excluded from metric attributes to avoid duplication. If the name_field isn't found, the metric will be dropped.

Response Format

The receiver expects JSON responses in one of two formats:

1. Top-level array:

[
  {"id": "1", "message": "log entry 1"},
  {"id": "2", "message": "log entry 2"}
]

1. Object with data field:

{
  "data": [
    {"id": "1", "message": "log entry 1"},
    {"id": "2", "message": "log entry 2"}
  ],
  "total": 2
}

When using the second format, specify the field name in response_field (e.g., "data").

Checkpointing

When a storage extension is configured, the receiver saves its pagination state to storage. This allows the receiver to resume from where it left off after a restart, preventing duplicate data collection.

The checkpoint includes:

• Current pagination state (offset/page number/timestamp)

• Number of pages fetched

For timestamp-based pagination, the timestamp is reset after each poll cycle to the initial timestamp, ensuring each poll starts fresh and only collects new data based on the time filter.