Transforms
editTransforms
editA transform processes the payload in the watch execution context to prepare the payload for watch actions.
If no transforms are defined, the actions have access to the payload as loaded by the watch input.
You can define transforms in two places:
- As a top level construct in the watch definition. In this case, the payload is transformed before any of the watch actions are executed.
- As part of the definition of a particular action. In this case, the payload is transformed before that action is executed. The transformation is only applied to the payload for that specific action.
If all actions require the same view of the payload, define a transform as part of the watch definition. If each action requires a different view of the payload, define different transforms as part of the action definitions so each action has the payload prepared by its own dedicated transform.
The following example defines two transforms, one at the watch level and one as part of the definition of the my_webhook
action.
Watch Transform Constructs.
{ "trigger" : { ...} "input" : { ... }, "condition" : { ... }, "transform" : { "search" : { "body" : { "query" : { "match_all" : {} } } } } "actions" : { "my_webhook": { "transform" : { "script" : "return ctx.payload.hits" } "webhook" : { "host" : "host.domain", "port" : 8089, "path" : "/notify/{{ctx.watch_id}}" } } ] ... }
Watcher supports three types of transforms: search
, script
and chain
.
Search Transform
editA Transform that executes a search on the cluster and replaces the current payload in the watch execution context with the returned search results. The following snippet shows how a simple search transform can be defined on the watch level:
Simple Search Transform.
{ ... "transform" : { "search" : { "request" : { "body" : { "query" : { "match_all" : {} }} } } } ... }
Like every other search based construct, one can make use of elasticsearch’s full search API by providing additional parameters:
Simple Search Transform.
{ "transform" : { "search" : { "request" : { "indices" : [ "logstash-*" ], "body" : { "size" : 0, "query" : { "match" : { "priority" : "error"} } } } } } }
The above example executes a count search over all the logstash indices, matching all
the events with error
priority.
The following table lists all available settings for the search transform:
Table 17. Search Transform Settings
Name | Required | Default | Description |
---|---|---|---|
|
no |
The search search type |
|
|
no |
all indices |
One or more indices to search on (may be a comma-delimited string or an array of indices names). |
|
no |
all types |
One or more document types to search on (may be a comma-delimited string or an array of document types names) |
|
no |
|
The body of the request. The request body follows the same structure you normally send in the body of a REST |
|
no |
|
Determines how to expand indices wildcards. Can be one of |
|
no |
|
A boolean value that determines whether the search should leniently ignore unavailable indices ((see multi-index support) |
|
no |
|
A boolean value that determines whether the search should leniently return no results when no indices are resolved ((see multi-index support) |
|
no |
- |
The body of the search template. See configure templates for more information. |
|
no |
30s |
The timeout for waiting for the search api call to return. If no response is returned within this time, the search transform times out and fails. This setting overrides the default internal search operations timeouts. |
Template Support
editAs can be seen in the table above, the search transform support mustache templates. This can either be as part of the body definition, or alternatively, point to a pre defined/registered template (either defined in a file or registered as a script in elasticsearch). The following snippet shows an example of a search that refers to the scheduled time of the watch:
Simple Search Transform using body template support.
{ "transform" : { "search" : { "index" : [ "logstash-*" ], "type" : "event", "body" : { "size" : 0, "query" : { "filtered" : { "filter" : { "bool" : { "must" : [ { "range" : { "@timestamp" : { "from" : "{{ctx.trigger.scheduled_time}}||-30s", "to" : "{{ctx.trigger.triggered_time}}" } } }, { "query" : { "match" : { "priority" : "error"} } } ] } } } } } } } }
The model of the template (based on which the mustache template will be evaluated) is a union between the provided
template.params
settings and the standard watch execution context model.
Simple Search Transform using an inline template.
{ "transform" : { "search" : { "index" : [ "logstash-*" ], "type" : "event", "body" : { "size" : 0, "template" { "inline" : { "query" : { "filtered" : { "filter" : { "bool" : { "must" : [ { "range" : { "@timestamp" : { "from" : "{{ctx.trigger.scheduled_time}}||-30s", "to" : "{{ctx.trigger.triggered_time}}" } } }, { "query" : { "match" : { "priority" : "{{priority}}"} } } ] } } } }, "params" : { "priority" : "error" } } } } } } }
Script Transform
editA Transform that executes a script on the current payload in the watch execution context and replaces it with a newly generated one. The following snippet shows how a simple script transform can be defined on the watch level:
Simple Script Transform.
A simple |
The executed script may either return a valid model that is the equivalent of a Java™ Map or a JSON object (you
will need to consult the documentation of the specific scripting language to find out what this construct is). Any
other value that is returned will be assigned and accessible to/via the _value
variable.
As seen above, the script
may hold a string value in which case it will be treated as the script itself and the default
elasticsearch script languages will be assumed (as described here). It
is possible to have more control over the scripting languages and also utilize pre-registered/pre-configured scripts
in elasticsearch. For this, the script
field will be defined as an object, and the following table lists the possible
settings that can be configured:
Table 18. Script Transform Settings
Name | Required | Default | Description |
---|---|---|---|
|
yes* |
- |
When using an inline script, this field holds the script itself. |
|
yes* |
- |
When refering to a script file, this field holds the name of the file. |
|
yes* |
- |
When refering to an indexed script, this field holds the id of the script. |
|
no |
|
The script language |
|
no |
- |
Additional parameters/variables that are accessible by the script |
-
When using the object notation of the script, one (and only one) of
inline
,file
orid
fields must be defined
In addition to the provided params
, the scripts also have access to the Standard Watch Execution Context Parameters
Script Type
editWhen suing inline
scripts, if you’re running Elasticsearch 1.3.8 or above, or 1.4.3 or above,
you will need to explicitly enable dynamic scripts
in elasticsearch.yml
.
As indicated by the table above, it is possible to utilize the full scripting support in elasticsearch and to base the script
on pre-registered indexed scripts or pre-defined scripts in file. Please note, for security reasons, starting from elasticsearch
v1.4.3
, inline groovy scripts are disabled by default. Furthermore, it is considered a best practice to pre-define the script
in stored files. To read more about elasticsearch search scripting support and possible related vulnerabilities, please
see here.
Chain Transform
editA Transform that executes an ordereed list of configured transforms in a chain, where
the output of one transform serves as the input of the next transform in the chain. The payload that is
accepted by this transform serves as the input of the first transform in the chain and the output of the last
transform in the chain is the output of the chain
transform as a whole.
You can use chain transforms to build more complex transforms out of the other available transforms. For example,
you can combine a search
transform and a script
transform,
as shown in the following snippet:
"transform" : { "chain" : [ { "search" : { "indices" : [ "logstash-*" ], "body" : { "size" : 0, "query" : { "match" : { "priority" : "error" } } } } }, { "script" : "return [ error_count : ctx.payload.hits.total ]" } ] }
The |
|
The first transform in the chain (in this case, a |
|
The second and final transform in the chain (in this case, a |
This example executes a count
search on the cluster to look for error
events. The
search results are then passed to the second script
transform. The script
transform
extracts the total hit count and assigns it to the error_count
field in a newly-generated payload.
This newly-generated payload is the output of the chain
transform and replaces the
payload in the watch execution context.