Modifying log body timestamps

This guide provides a step-by-step process to modify timestamps within log bodies to ensure correct ingestion. The example used here demonstrates converting timestamps to a UTC-3 timezone.

To update the timestamp in the log body, four processors are required.

The four processors each manage a step in the overall process:

1. Extract the timestamp text and surrounding components from the log body.

2. Parse the extracted timestamp text into a timestamp object using strptime and provide a timezone.

3. Convert the timestamp object back into the log's original timestamp text format using ctime.

4. Combine the updated timestamp text with the original log components.

It can be helpful to think of the specific fields involved in each step:

1. body → attribute.pre_ts, attribute.ts, attribute.post_ts

2. attribute.ts → log.time

3. log.time → attribute.new_ts

4. attribute.pre_ts + attribute.new_ts + attribute.post_ts → body

Final result of updating the Body's timestamp

Extract timestamp from body

The first processor needed is . Select Body for the Source Field Type . Then select Attribute for the Target Field Type. Create a regex pattern to extract the timestamp and surrounding elements from the body using named capture groups.

Sep   8 14:57:32 asdfasdfasdf syslog message

(?P<pre_ts>^.*)(?P<ts>(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}\s\d{2}:\d{2}:\d{2})(?P<post_ts>.*$)

This regex separates the body into three attribute fields pre_ts (anything before the timestamp), ts (the timestamp), and post_ts (anything after the timestamp). The pre_ts and post_ts attributes are needed to reconstruct the body with a modified timestamp based on ts in the final processor described later.

The log body in our example starts with the timestamp, therefore pre_ts is blank.

Parse timestamp text into timestamp object

For the second processor we will add . Which parses attribute.ts and updates log.time.

Set the extracted timestamp from the last step as the Source Field. Select Attribute for the Log Field Type. Then type ts for the Source Field.

Choose Manual for the Log Time Format. The value of Timestamp Layout will depend on the logs being collected. You can often ask AI to produce the strptime layout based on an example.

Sep   8 14:57:32

%b  %e %H:%M:%S

ctime and strptime are often used interchangeably. ctime converts a Time object to a string, while strptime converts a string to a Time object.

We can now account for the missing timezone context. In this example, the log's timestamp is UTC-03:00, corresponding to the TZ Identifier America/Argentina/Buenos_Aires. We will assign this as the value of Location.

You can specify a timezone for Location using a TZ Identifier only if Log Time Format is set to Manual

log.time now accurately reflects the original log's timestamp as UTC, displaying the expected 3-hour difference.

Make the modified timestamp text

We will now utilize the processor to generate a field attribute.new_ts. This field's value will convert log.time into a UTC timestamp while maintaining the original log's format.

Select Attributes for the Target Field Type. Then type new_ts for the Target Field. The Timestamp Format will match the strptime layout used in the last step, unless there is a specific need to modify the timestamp format.

At this stage new_ts will match the original format but account for the timezone

Overwrite the body with the modified timestamp

Finally we need to rebuild log body with the modified timestamp by using .

Choose Attributes under Source Field Type. Then enter each of the following for Source Fields: pre_ts, new_ts, and post_ts. Finally Select Body for Target Field Type.

Body should match the original but with your corrected timestamp