IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Configuration options Test your regexp pattern for multiline »

› › › › ›

Examples of multiline configuration

edit

Examples of multiline configuration

edit

The examples in this section cover the following use cases:

Combining a Java stack trace into a single event
Combining C-style line continuations into a single event
Combining multiple lines from time-stamped events

Java stack traces

edit

Java stack traces consist of multiple lines, with each line after the initial line beginning with whitespace, as in this example:

Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

This configuration merges any line that begins with whitespace up to the previous line:

multiline:
  type: pattern
  pattern: '^\s'
  negate: false
  match: after

This is a slightly more complex Java stack trace example:

Exception in thread "main" java.lang.IllegalStateException: A book has a null property
       at com.example.myproject.Author.getBookIds(Author.java:38)
       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
       at com.example.myproject.Book.getId(Book.java:22)
       at com.example.myproject.Author.getBookIds(Author.java:35)
       ... 1 more

To consolidate these lines into a single event, use the following multiline configuration:

multiline:
  type: pattern
  pattern: '^\s+(at|.{3})\s+\\b|^Caused by:'
  negate: false
  match: after

In this example, the pattern matches and merges the following lines: - a line that begins with spaces followed by the word at or ... - a line that begins with the words Caused by:

In Python’s string literals, \b is the backspace character (ASCII value 8). As raw strings are not used, Python would convert the \b to a backspace. In order for our regular expression to match as expected, you need to escape the backslash \ in \b to \\b, which will produce the correct regular expression upon compiling.

Line continuations

edit

Several programming languages use the backslash (\) character at the end of a line to denote that the line continues, as in this example:

printf ("%10.10ld  \t %10.10ld \t %s\
  %f", w, x, y, z );

To consolidate these lines into a single event, use the following multiline configuration:

multiline:
  type: pattern
  pattern: '\\\\$'
  negate: false
  match: after

This configuration merges any line that ends with the \ character with the line that follows it.

Note that you should escape the opening backslash (\) twice in the regular expression, as raw strings are not used. Thus, \\\\$ will produce the required regular expression upon compiling.

Timestamps

edit

Activity logs from services such as Elasticsearch typically begin with a timestamp, followed by information on the specific activity, as in this example:

[2015-08-24 11:49:14,389][INFO ][env                      ] [Letha] using [1] data paths, mounts [[/
(/dev/disk1)]], net usable_space [34.5gb], net total_space [118.9gb], types [hfs]

To consolidate these lines into a single event, use the following multiline configuration:

multiline:
  type: pattern
  pattern: '^\\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  negate: true
  match: after

This configuration uses the negate: true and match: after settings to specify that any line that does not match the specified pattern belongs to the previous line.

Note that you should escape the opening square bracket ([) in the regular expression, because it specifies a character class i.e. a set of characters that you wish to match. You also have to escape the backslash (\) used for escaping the opening square bracket as raw strings are not used. Thus, ^\\[ will produce the required regular expression upon compiling.

Application events

edit

Sometimes your application logs contain events, that begin and end with custom markers, such as the following example:

[2015-08-24 11:49:14,389] Start new event
[2015-08-24 11:49:14,395] Content of processing something
[2015-08-24 11:49:14,399] End event

To consolidate these lines into a single event, use the following multiline configuration:

multiline:
  type: pattern
  pattern: 'Start new event'
  negate: true
  match: after
  flush_pattern: 'End event'

The flush_pattern option specifies a regex at which the current multiline will be flushed. If you think of the pattern option specifying the beginning of an event, the flush_pattern option will specify the end or last line of the event.

This example will not work correctly if start/end log blocks are mixed with non-multiline logs, or if different start/end log blocks overlap with each other. For instance, Some other log log lines in the following example will be merged into a single multiline document because they neither match inputs.[].multiline.pattern nor inputs.[].multiline.flush_pattern, and inputs.[].multiline.negate is set to true.

[2015-08-24 11:49:14,389] Start new event
[2015-08-24 11:49:14,395] Content of processing something
[2015-08-24 11:49:14,399] End event
[2015-08-24 11:50:14,389] Some other log
[2015-08-24 11:50:14,395] Some other log
[2015-08-24 11:50:14,399] Some other log
[2015-08-24 11:51:14,389] Start new event
[2015-08-24 11:51:14,395] Content of processing something
[2015-08-24 11:51:14,399] End event

« Configuration options Test your regexp pattern for multiline »