Modifying log body timestamps

To update the timestamp in the log body, four processors are required.

The four processors each manage a step in the overall process:

1. Extract the timestamp text and surrounding components from the log body.

2. Parse the extracted timestamp text into a timestamp object using strptime and provide a timezone.

3. Convert the timestamp object back into the log's original timestamp text format using ctime.

4. Combine the updated timestamp text with the original log components.

It can be helpful to think of the specific fields involved in each step:

1. body → attribute.pre_ts, attribute.ts, attribute.post_ts

2. attribute.ts → log.time

3. log.time → attribute.new_ts

4. attribute.pre_ts + attribute.new_ts + attribute.post_ts → body

Extract timestamp from body

The first processor needed is Parse with Regex. Select Body for the Source Field Type . Then select Attribute for the Target Field Type. Create a regex pattern to extract the timestamp and surrounding elements from the body using named capture groups.

Sep   8 14:57:32 asdfasdfasdf syslog message

(?P<pre_ts>^.*)(?P<ts>(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}\s\d{2}:\d{2}:\d{2})(?P<post_ts>.*$)

This regex separates the body into three attribute fields pre_ts (anything before the timestamp), ts (the timestamp), and post_ts (anything after the timestamp). The pre_ts and post_ts attributes are needed to reconstruct the body with a modified timestamp based on ts in the final processor described later.

Parse timestamp text into timestamp object

For the second processor we will add Parse Timestamp. Which parses attribute.ts and updates log.time.

Set the extracted timestamp from the last step as the Source Field. Select Attribute for the Log Field Type. Then type ts for the Source Field.

Choose Manual for the Log Time Format. The value of Timestamp Layout will depend on the logs being collected. You can often ask AI to produce the strptime layout based on an example.

Sep   8 14:57:32

%b  %e %H:%M:%S

We can now account for the missing timezone context. In this example, the log's timestamp is UTC-03:00, corresponding to the TZ Identifier America/Argentina/Buenos_Aires. We will assign this as the value of Location.

Make the modified timestamp text

We will now utilize the Rewrite Timestamp processor to generate a field attribute.new_ts. This field's value will convert log.time into a UTC timestamp while maintaining the original log's format.

Select Attributes for the Target Field Type. Then type new_ts for the Target Field. The Timestamp Format will match the strptime layout used in the last step, unless there is a specific need to modify the timestamp format.

Overwrite the body with the modified timestamp

Finally we need to rebuild log body with the modified timestamp by using Concat.

Choose Attributes under Source Field Type. Then enter each of the following for Source Fields: pre_ts, new_ts, and post_ts. Finally Select Body for Target Field Type.