Log Processors

Every pipeline includes a chain of processors that define the transformations it will apply to logs. When a log matches a pipeline's filter, it is transformed by each processor in the pipeline one by one.

The following log transformation processors are available for defining pipelines.

Regex

The Regex processor can be used to extract information out of text using regular expressions.

Processor Fields

FieldDescription
NameA descriptive name for the processor.
PatternThe regex pattern to be used. Must include atleast one named capture group
Parse FromThe log text field to parse from. Eg: body or attributes.sessionInfo
Parse ToThe path to parse to. Eg: If set to attributes, a capture group like (?P<userId>.+) in the regex pattern would get stored in attributes.userId
On ErrorWhat to do if the processor fails. Options are to drop the log or send it to the next processor

Grok

The Grok processor can be used to extract information out of text using grok patterns.
Grok is a regular expression dialect with convenient aliases for commonly used expressions.

Processor Fields

FieldDescription
NameA descriptive name for the processor.
PatternThe grok pattern to be used. Must include atleast one named capture group
Parse FromThe log text field to parse from. Eg: body or attributes.sessionInfo
Parse ToThe path to parse to. Eg: If set to attributes, a capture group like %{WORD:userId} in the grok pattern would get stored in attributes.userId
On ErrorWhat to do if the processor fails. Options are to drop the log or send it to the next processor
📝 Note

By default, values extracted using grok patterns are strings.
For example, parsing status: 202 with the pattern status: %{INT:status_code} will extract status_code as a string with value "202".

However, it is possible to extract float or int typed values by adding a 3rd part to grok capture groups.
For example, parsing status: 202 with the pattern status: %{INT:status_code:int} will extract status_code as an integer with value 202.

This can enable the use of numeric operators (>, < etc) on the extracted values and unlock features like using the values as metrics in dashboards.

JSON Parser

The JSON parsing processor can be used to parse serialized JSON text into log attributes.

Processor Fields

FieldDescription
NameA descriptive name for the processor.
Parse FromThe log field containing serialized JSON text. Eg: body or attributes.sessionInfo
Parse ToThe path to parse to. Eg: If set to attributes, parsing the JSON text '{ "userId": 8888 }' would set attributes.userId to 8888

Trace Parser

The trace processor can be used to populate trace id, span id and trace flags for a log.
Populating trace identifiers in logs allows navigation to and from corresponding traces for correlation.

Processor Fields

FieldDescription
NameA descriptive name for the processor.
Parse Trace Id FromThe log field containing otel Trace Id. Eg: attributes.myTraceId
Value at the specified path must be an even length string of hex characters
Parse Span Id FromThe log field containing otel Span Id. Eg: attributes.mySpanId
Value at the specified path must be an even length string of hex characters
Parse Trace Flags FromThe log field containing otel Trace Flags. Eg: attributes.myTraceFlags
Value at the specified path must be an unsigned int
📝 Note

At least one field among Parse Trace Id From, Parse Span Id From and Parse Trace Flags From must be specified.

Timestamp Parser

The timestamp parsing processor can be used to parse log timestamp out of a log field.

Processor Fields

FieldDescription
Name of Timestamp Parsing ProcessorA descriptive name for the processor.
Parse Timestamp Value FromThe log field containing timestamp value. Eg: attributes.timestamp
Timestamp Format TypeType of timestamp value to be parsed.
epoch can be used for parsing unix time and strptime can be used for parsing human readable values using ctime-like directives such as %Y (4-digit year) and %H (2-digit hour).
Timestamp FormatFormat for parsing timestamp value.
For example %Y-%m-%d can be used for parsing values like 2023-12-06 when Timestamp Format Type is strptime, or seconds.milliseconds can be used for parsing unix time values like 1701869406.245 when Timestamp Format Type is epoch

Severity Parser

The severity parsing processor can be used to parse log severity out of a log field.

Processor Fields

FieldDescription
Name of Severity Parsing ProcessorA descriptive name for the processor.
Parse Severity Value FromThe log field to parse severity from. For example attributes.log_level
Values for level TRACEComma separated list of values that should be mapped to level TRACE. For example 0, trace
Values for level DEBUGComma separated list of values that should be mapped to level DEBUG. For example debug, 2xx
Values for level INFOComma separated list of values that should be mapped to level INFO. For example info, 3xx
Values for level WARNComma separated list of values that should be mapped to level WARN. For example warning, 4xx
Values for level ERRORComma separated list of values that should be mapped to level ERROR. For example error, 5xx
Values for level FATALComma separated list of values that should be mapped to level FATAL. For example panic, -1
📝 Note

Severity level values are case insensitive.
As a special case, 2xx, 3xx, 4xx and 5xx can be used to map number ranges to severity levels. This can be useful for mapping HTTP status codes. For example 5xx can be used to parse numbers between 500-599 into level ERROR.

Add

The add processor can be used to add a field to the log.

Processor Fields

FieldDescription
NameA descriptive name for the processor.
FieldPath of the field to be added. Must be of the form attributes.* or resource.*
ValueThe value to be set in the specified field
📝 Note

The value field can be set to an expression which will get evaluated for each entry to set the value.
For example the value can be set to EXPR(attributes.subtotal + attributes.taxes) to add a new field for total.
Value expressions are also useful for accessing array items that can't be referenced with field paths in operators like COPY and MOVE, for example EXPR(attributes.locale[0]).

Remove

The remove processor can be used for removing unwanted log fields such as PII.

Processor Fields

FieldDescription
NameA descriptive name for the processor.
FieldPath of the field to be removed. Must be of the form attributes.* or resource.*

Move

The move processor can be used to move or rename a log field.

Processor Fields

FieldDescription
NameA descriptive name for the processor.
FromPath of the field to be moved. Must be of the form attributes.* or resource.*
ToPath to move the field to. Must be of the form attributes.* or resource.*

Copy

The copy processor can be used to copy log fields.

Processor Fields

FieldDescription
NameA descriptive name for the processor.
FromPath of the field to be copied. Must be of the form attributes.* or resource.*
ToPath to copy the field to. Must be of the form attributes.* or resource.*

Was this page helpful?