New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« EQL search EQL function reference »
Elastic Docs ›Elasticsearch Guide [7.10] ›EQL search

EQL syntax reference

edit

EQL syntax reference

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

Basic syntax

edit

EQL queries require an event category and a matching condition. The where keyword connects them.

event_category where condition

For example, the following EQL query matches process events with a process.name field value of svchost.exe:

process where process.name == "svchost.exe"

Event categories

edit

An event category is a valid, indexed value of the event category field. You can set the event category field using the event_category_field parameter of the EQL search API.

Match any event category

edit

To match events of any category, use the any keyword. You can also use the any keyword to search for documents without a event category field.

For example, the following EQL query matches any documents with a network.protocol field value of http:

any where network.protocol == "http"

Escape an event category

edit

Use enclosing double quotes (") or three enclosing double quotes (""") to escape event categories that:

  • Contain a special character, such as a hyphen (-) or dot (.)
  • Contain a space
  • Start with a numeral 
".my.event.category"
"my-event-category"
"my event category"
"6eventcategory"

""".my.event.category"""
"""my-event-category"""
"""my event category"""
"""6eventcategory"""

Escape a field name

edit

Use enclosing enclosing backticks (`) to escape field names that:

  • Contain a hyphen (-)
  • Contain a space
  • Start with a numeral 
`my-field`
`my field`
`6myfield`

Use double backticks (``) to escape any backticks (`) in the field name.

my`field -> `my``field`

Conditions

edit

A condition consists of one or more criteria an event must match. You can specify and combine these criteria using the following operators. Most EQL operators are case-sensitive by default.

Comparison operators

edit
<   <=   ==   :   !=   >=   >
< (less than)
Returns true if the value to the left of the operator is less than the value to the right. Otherwise returns false.
<= (less than or equal)
Returns true if the value to the left of the operator is less than or equal to the value to the right. Otherwise returns false.
== (equal, case-sensitive)
Returns true if the values to the left and right of the operator are equal. Otherwise returns false. For strings, matching is case-sensitive.
: (equal, case-insensitive)
Returns true if strings to the left and right of the operator are equal. Otherwise returns false. Matching is case-insensitive and can only be used to compare strings.

Avoid using the == or : operators to perform exact matching on text field values.

By default, Elasticsearch changes the values of text fields as part of analysis. This can make finding exact matches for text field values difficult.

To search text fields, consider using a query DSL filter that contains a match query.

!= (not equal, case-sensitive)
Returns true if the values to the left and right of the operator are not equal. Otherwise returns false. For strings, matching is case-sensitive.
>= (greater than or equal)
Returns true if the value to the left of the operator is greater than or equal to the value to the right. Otherwise returns false. When comparing strings, the operator uses a case-sensitive lexicographic order.
> (greater than)
Returns true if the value to the left of the operator is greater than the value to the right. Otherwise returns false. When comparing strings, the operator uses a case-sensitive lexicographic order.

= is not supported as an equal operator. Use == or : instead.

You cannot chain comparison operators. Instead, use a logical operator between comparisons. For example, foo < bar <= baz is not supported. However, you can rewrite the expression as foo < bar and bar <= baz, which is supported.

You also cannot use comparison operators to compare a field to another field. This applies even if the fields are changed using a function.

Example
The following EQL query compares the process.parent_name field value to a static value, foo. This comparison is supported.

However, the query also compares the process.parent.name field value to the process.name field. This comparison is not supported and will return an error for the entire query.

process where process.parent.name == "foo" and process.parent.name == process.name

Instead, you can rewrite the query to compare both the process.parent.name and process.name fields to static values.

process where process.parent.name == "foo" and process.name == "foo"

Logical operators

edit
and  or  not
and
Returns true only if the condition to the left and right both return true. Otherwise returns false.
or
Returns true if one of the conditions to the left or right true. Otherwise returns false.
not
Returns true if the condition to the right is false.

Lookup operators

edit
user.name in ("Administrator", "SYSTEM", "NETWORK SERVICE")
user.name not in ("Administrator", "SYSTEM", "NETWORK SERVICE")
in (case-sensitive)
Returns true if the value is contained in the provided list. For strings, matching is case-sensitive.
not in (case-sensitive)
Returns true if the value is not contained in the provided list. For strings, matching is case-sensitive.

Math operators

edit
+  -  *  /  %
+ (add)
Adds the values to the left and right of the operator.
- (subtract)
Subtracts the value to the right of the operator from the value to the left.
* (multiply)
Multiplies the values to the left and right of the operator.
/ (divide)

Divides the value to the left of the operator by the value to the right.

If both the dividend and divisor are integers, the divide (\) operation rounds down any returned floating point numbers to the nearest integer. To avoid rounding, convert either the dividend or divisor to a float.

Example
The process.args_count field is a long integer field containing a count of process arguments.

A user might expect the following EQL query to only match events with a process.args_count value of 4.

process where ( 4 / process.args_count ) == 1

However, the EQL query matches events with a process.args_count value of 3 or 4.

For events with a process.args_count value of 3, the divide operation returns a float of 1.333..., which is rounded down to 1.

To match only events with a process.args_count value of 4, convert either the dividend or divisor to a float.

The following EQL query changes the integer 4 to the equivalent float 4.0.

process where ( 4.0 / process.args_count ) == 1
% (modulo)
Divides the value to the left of the operator by the value to the right. Returns only the remainder.

Match any condition

edit

To match events solely on event category, use the where true condition.

For example, the following EQL query matches any file events:

file where true

To match any event, you can combine the any keyword with the where true condition:

any where true

Strings

edit

Strings are enclosed in double quotes (").

"hello world"

Strings enclosed in single quotes (') are not supported.

Escape characters in a string

edit

When used within a string, special characters, such as a carriage return or double quote ("), must be escaped with a preceding backslash (\).

"example \r of \" escaped \n characters"
Escape sequence Literal character

\n

A newline (linefeed) character

\r

A carriage return character

\t

A tab character

\\

A backslash (\) character

\"

A double quote (") character

The single quote (') character is reserved for future use. You cannot use an escaped single quote (\') for literal strings. Use an escaped double quote (\") instead.

Raw strings

edit

Raw strings treat special characters, such as backslashes (\), as literal characters. Raw strings are enclosed in three double quotes (""").

"""Raw string with a literal double quote " and blackslash \ included"""

A raw string cannot contain three consecutive double quotes ("""). Instead, use a regular string with the \" escape sequence.

"String containing \"\"\" three double quotes"

Sequences

edit

You can use EQL sequences to describe and match an ordered series of events. Each item in a sequence is an event category and event condition, surrounded by square brackets ([ ]). Events are listed in ascending chronological order, with the most recent event listed last.

sequence
  [ event_category_1 where condition_1 ]
  [ event_category_2 where condition_2 ]
  ...

Example
The following EQL sequence query matches this series of ordered events:

  1. Start with an event with:

    • An event category of file
    • A file.extension of exe
  2. Followed by an event with an event category of process 
sequence
  [ file where file.extension == "exe" ]
  [ process where true ]

with maxspan keywords

edit

You can use the with maxspan keywords to constrain a sequence to a specified timespan. All events in a matching sequence must occur within this duration, starting at the first event’s timestamp.

The maxspan keyword accepts time value arguments.

sequence with maxspan=30s
  [ event_category_1 where condition_1 ] by field_baz
  [ event_category_2 where condition_2 ] by field_bar
  ...

Example
The following sequence query uses a maxspan value of 15m (15 minutes). Events in a matching sequence must occur within 15 minutes of the first event’s timestamp.

sequence with maxspan=15m
  [ file where file.extension == "exe" ]
  [ process where true ]

by keyword

edit

You can use the by keyword with sequences to only match events that share the same field values. If a field value should be shared across all events, you can use sequence by.

sequence by field_foo
  [ event_category_1 where condition_1 ] by field_baz
  [ event_category_2 where condition_2 ] by field_bar
  ...

Example
The following sequence query uses the by keyword to constrain matching events to:

  • Events with the same user.name value
  • file events with a file.path value equal to the following process event’s process.path value. 
sequence
  [ file where file.extension == "exe" ] by user.name, file.path
  [ process where true ] by user.name, process.path

Because the user.name field is shared across all events in the sequence, it can be included using sequence by. The following sequence is equivalent to the prior one.

sequence by user.name
  [ file where file.extension == "exe" ] by file.path
  [ process where true ] by process.path

You can combine the sequence by and with maxspan keywords to constrain a sequence by both field values and a timespan.

sequence by field_foo with maxspan=30s
  [ event_category_1 where condition_1 ] by field_baz
  [ event_category_2 where condition_2 ] by field_bar
  ...

Example
The following sequence query uses the sequence by keyword and with maxspan keywords to match only a sequence of events that:

  • Share the same user.name field values
  • Occur within 15m (15 minutes) of the first matching event 
sequence by user.name with maxspan=15m
  [ file where file.extension == "exe" ] by file.path
  [ process where true ] by process.path

until keyword

edit

You can use the until keyword to specify an expiration event for a sequence. If this expiration event occurs between matching events in a sequence, the sequence expires and is not considered a match. If the expiration event occurs after matching events in a sequence, the sequence is still considered a match. The expiration event is not included in the results.

sequence
  [ event_category_1 where condition_1 ]
  [ event_category_2 where condition_2 ]
  ...
until [ event_category_3 where condition_3 ]

Example
A dataset contains the following event sequences, grouped by shared IDs:

A, B
A, B, C
A, C, B

The following EQL query searches the dataset for sequences containing event A followed by event B. Event C is used as an expiration event.

sequence by ID
  A
  B
until C

The query matches sequences A, B and A, B, C but not A, C, B.

The until keyword can be useful when searching for process sequences in Windows event logs.

In Windows, a process ID (PID) is unique only while a process is running. After a process terminates, its PID can be reused.

You can search for a sequence of events with the same PID value using the by and sequence by keywords.

Example
The following EQL query uses the sequence by keyword to match a sequence of events that share the same process.pid value.

sequence by process.pid
  [ process where event.type == "start" and process.name == "cmd.exe" ]
  [ process where file.extension == "exe" ]

However, due to PID reuse, this can result in a matching sequence that contains events across unrelated processes. To prevent false positives, you can use the until keyword to end matching sequences before a process termination event.

The following EQL query uses the until keyword to end sequences before process events with an event.type of stop. These events indicate a process has been terminated.

sequence by process.pid
  [ process where event.type == "start" and process.name == "cmd.exe" ]
  [ process where file.extension == "exe" ]
until [ process where event.type == "stop" ]

Functions

edit

You can use EQL functions to convert data types, perform math, manipulate strings, and more. Most functions are case-sensitive by default.

For a list of supported functions, see Function reference.

Using functions in EQL queries can result in slower search speeds. If you often use functions to transform indexed data, you can speed up search by making these changes during indexing instead. However, that often means slower index speeds.

Example
An index contains the file.path field. file.path contains the full path to a file, including the file extension.

When running EQL searches, users often use the endsWith function with the file.path field to match file extensions:

file where endsWith(file.path,".exe") or endsWith(file.path,".dll")

While this works, it can be repetitive to write and can slow search speeds. To speed up search, you can do the following instead:

  1. Add a new field, file.extension, to the index. The file.extension field will contain only the file extension from the file.path field.
  2. Use an ingest pipeline containing the grok processor or another preprocessor tool to extract the file extension from the file.path field before indexing.
  3. Index the extracted file extension to the file.extension field.

These changes may slow indexing but allow for faster searches. Users can use the file.extension field instead of multiple endsWith function calls:

file where file.extension in ("exe", "dll")

We recommend testing and benchmarking any indexing changes before deploying them in production. See Tune for indexing speed and Tune for search speed.

Pipes

edit

EQL pipes filter, aggregate, and post-process events returned by an EQL query. You can use pipes to narrow down EQL query results or make them more specific.

Pipes are delimited using the pipe (|) character.

event_category where condition | pipe

Example
The following EQL query uses the tail pipe to return only the 10 most recent events matching the query.

authentication where agent.id == 4624
| tail 10

You can pass the output of a pipe to another pipe. This lets you use multiple pipes with a single query.

For a list of supported pipes, see Pipe reference.

Limitations

edit

EQL does not support the following features and syntax.

Comparing fields

edit

You cannot use EQL comparison operators to compare a field to another field. This applies even if the fields are changed using a function.

Array field values are not supported

edit

EQL does not support array field values, also known as multi-value fields. EQL searches on array field values may return inconsistent results.

EQL search on nested fields

edit

You cannot use EQL to search the values of a nested field or the sub-fields of a nested field. However, data streams and indices containing nested field mappings are otherwise supported.

Differences from Endgame EQL syntax

edit

Elasticsearch EQL differs from the Elastic Endgame EQL syntax as follows:

  • Most operators and functions in Elasticsearch EQL are case-sensitive. For case-insensitive equality comparisons, use the : operator.
  • Comparisons using the == and != operators do not expand wildcard characters. For example, process_name == "cmd*.exe" interprets * as a literal asterisk, not a wildcard. For case-sensitive wildcard matching, use the wildcard function.
  • = cannot be substituted for the == operator.
  • Strings enclosed in single quotes (') are not supported. Enclose strings in double quotes (") instead.
  • ?" and ?' do not indicate raw strings. Enclose raw strings in three double quotes (""") instead.

  • Elasticsearch EQL does not support:

How sequence queries handle matches

edit

Sequence queries don’t find all potential matches for a sequence. This approach would be too slow and costly for large event data sets. Instead, a sequence query handles pending sequence matches as a state machine:

  • Each event item in the sequence query is a state in the machine.
  • Only one pending sequence can be in each state at a time.
  • If two pending sequences are in the same state at the same time, the most recent sequence overwrites the older one.
  • If the query includes by fields, the query uses a separate state machine for each unique by field value.
Example

A data set contains the following process events in ascending chronological order:

{ "index" : { "_id": "1" } }
{ "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
{ "index" : { "_id": "2" } }
{ "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
{ "index" : { "_id": "3" } }
{ "user": { "name": "elkbee" }, "process": { "name": "bash" }, ...}
{ "index" : { "_id": "4" } }
{ "user": { "name": "root" }, "process": { "name": "bash" }, ...}
{ "index" : { "_id": "5" } }
{ "user": { "name": "root" }, "process": { "name": "bash" }, ...}
{ "index" : { "_id": "6" } }
{ "user": { "name": "elkbee" }, "process": { "name": "attrib" }, ...}
{ "index" : { "_id": "7" } }
{ "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
{ "index" : { "_id": "8" } }
{ "user": { "name": "elkbee" }, "process": { "name": "bash" }, ...}
{ "index" : { "_id": "9" } }
{ "user": { "name": "root" }, "process": { "name": "cat" }, ...}
{ "index" : { "_id": "10" } }
{ "user": { "name": "elkbee" }, "process": { "name": "cat" }, ...}
{ "index" : { "_id": "11" } }
{ "user": { "name": "root" }, "process": { "name": "cat" }, ...}

An EQL sequence query searches the data set:

sequence by user.name
  [process where process.name == "attrib"]
  [process where process.name == "bash"]
  [process where process.name == "cat"]

The query’s event items correspond to the following states:

  • State A: [process where process.name == "attrib"]
  • State B: [process where process.name == "bash"]
  • Complete: [process where process.name == "cat"]
sequence state machine

To find matching sequences, the query uses separate state machines for each unique user.name value. Based on the data set, you can expect two state machines: one for the root user and one for elkbee.

separate state machines

Pending sequence matches move through each machine’s states as follows:

{ "index" : { "_id": "1" } }
{ "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
// Creates sequence [1] in state A for the "root" user.
//
// +------------------------"root"------------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+     +------------+  |
// |  |    [1]    |     |           |     |            |  |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "2" } }
{ "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
// Creates sequence [2] in state A for "root", overwriting sequence [1].
//
// +------------------------"root"------------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+     +------------+  |
// |  |    [2]    |     |           |     |            |  |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "3" } }
{ "user": { "name": "elkbee" }, "process": { "name": "bash" }, ...}
// Nothing happens. The "elkbee" user has no pending sequence to move
// from state A to state B.
//
// +-----------------------"elkbee"-----------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+     +------------+  |
// |  |           |     |           |     |            |  |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "4" } }
{ "user": { "name": "root" }, "process": { "name": "bash" }, ...}
// Sequence [2] moves out of state A for "root".
// State B for "root" now contains [2, 4].
// State A for "root" is empty.
//
// +------------------------"root"------------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+ --> +-----------+     +------------+  |
// |  |           |     |   [2, 4]  |     |            |  |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "5" } }
{ "user": { "name": "root" }, "process": { "name": "bash" }, ...}
// Nothing happens. State A is empty for "root".
//
// +------------------------"root"------------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+     +------------+  |
// |  |           |     |   [2, 4]  |     |            |  |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "6" } }
{ "user": { "name": "elkbee" }, "process": { "name": "attrib" }, ...}
// Creates sequence [6] in state A for "elkbee".
//
// +-----------------------"elkbee"-----------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+     +------------+  |
// |  |    [6]    |     |           |     |            |  |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "7" } }
{ "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
// Creates sequence [7] in state A for "root".
// Sequence [2, 4] remains in state B for "root".
//
// +------------------------"root"------------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+     +------------+  |
// |  |    [7]    |     |   [2, 4]  |     |            |  |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "8" } }
{ "user": { "name": "elkbee" }, "process": { "name": "bash" }, ...}
// Sequence [6, 8] moves to state B for "elkbee".
// State A for "elkbee" is now empty.
//
// +-----------------------"elkbee"-----------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+ --> +-----------+     +------------+  |
// |  |           |     |   [6, 8]  |     |            |  |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "9" } }
{ "user": { "name": "root" }, "process": { "name": "cat" }, ...}
// Sequence [2, 4, 9] is complete for "root".
// State B for "root" is now empty.
// Sequence [7] remains in state A.
//
// +------------------------"root"------------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+ --> +------------+  |
// |  |    [7]    |     |           |     |  [2, 4, 9] |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "10" } }
{ "user": { "name": "elkbee" }, "process": { "name": "cat" }, ...}
// Sequence [6, 8, 10] is complete for "elkbee".
// State A and B for "elkbee" are now empty.
//
// +-----------------------"elkbee"-----------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+ --> +------------+  |
// |  |           |     |           |     | [6, 8, 10] |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+

{ "index" : { "_id": "11" } }
{ "user": { "name": "root" }, "process": { "name": "cat" }, ...}
// Nothing happens.
// The machines for "root" and "elkbee" remain the same.
//
// +------------------------"root"------------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+     +------------+  |
// |  |    [7]    |     |           |     |  [2, 4, 9] |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+
//
// +-----------------------"elkbee"-----------------------+
// |  +-----------+     +-----------+     +------------+  |
// |  |  State A  |     |  State B  |     |  Complete  |  |
// |  +-----------+     +-----------+     +------------+  |
// |  |           |     |           |     | [6, 8, 10] |
// |  +-----------+     +-----------+     +------------+  |
// +------------------------------------------------------+
« EQL search EQL function reference »