Variables and conditions in input configurations

edit

Variables and conditions in input configurations

edit

When running Elastic Agent in some environments, you might not know all the input configuration details up front. To solve this problem, the input configuration accepts variables and conditions that get evaluated at runtime using information from the running environment. Similar to autodiscovery, these capabilities allow you to apply configurations dynamically.

Let’s consider a unique agent policy that is deployed on two machines: a Linux machine named "linux-app" and a Windows machine named "winapp". Notice that the configuration has some variable references: ${host.name} and ${host.platform}:

inputs:
 - type: logfile
   streams:
    - paths: /var/log/${host.name}/another.log
      condition: ${host.platform} == "linux"
    - path: c:/service/app.log
      condition: ${host.platform} == "windows"

At runtime, Elastic Agent resolves variables and evaluates the conditions based on values provided by the environment, generating two possible input configurations.

On the Windows machine:

inputs:
 - type: logfile
   streams:
    - path: c:/service/app.log

On the Linux machine:

inputs:
 - type: logfile
   streams:
    - paths: /var/log/linux-app/another.log

Using variable substitution along with conditions allows you to create concise, but flexible input configurations that adapt to their deployed environment.

Variable substitution

edit

The syntax for variable substitution is ${var}, where var is the name of a variable defined by a provider. A provider defines key/value pairs that are used for variable substitution and conditions.

Elastic Agent supports a variety of providers, such as host and local, that supply variables to Elastic Agent. For example, earlier you saw ${host.name} used to resolve the path to the host’s log file based on the {host.platform} value. Both of these values were provided by the host provider.

All providers are enabled by default when Elastic Agent starts. If a provider cannot be configured, its variables are ignored.

See Providers for more detail.

The following agent policy uses a custom key named foo to resolve a value defined by a local provider:

inputs:
 - type: logfile
   streams:
    - paths: /var/log/${foo}/another.log

providers:
  local:
    vars:
      foo: bar

The policy generated by this configuration looks like this:

inputs:
 - type: logfile
   streams:
    - paths: /var/log/bar/another.log

When an input uses a variable substitution that is not present in the current key/value mappings being evaluated, the input is removed in the result.

For example, this agent policy uses an unknown key:

inputs:
  - type: logfile
    path: "/var/log/foo"
  - type: logfile
    path: "${ unknown.key }"

The policy generated by this configuration looks like this:

inputs:
  - type: logfile
    path: "/var/log/foo"
Alternative variables and constants
edit

Variable substitution can also define alternative variables or a constant.

To define a constant, use either ' or ". When a constant is reached during variable evaluation, any remaining variables are ignored, so a constant should be the last entry in the substitution.

To define alternatives, use | followed by the next variable or constant. The power comes from allowing the input to define the preference order of the substitution by chaining multiple variables together.

For example, the following agent policy chains together multiple variables to set the log path based on information provided by the running container environment. The constant /var/log/other is used to end of the path, which is common to both providers:

inputs:
  - type: logfile
    path: "/var/log/foo"
  - type: logfile
    path: "${docker.paths.log|kubernetes.container.paths.log|'/var/log/other'}"

Providers

edit

Providers supply the key/values pairs that are used for variable substitution and conditionals. Each provider’s keys are automatically prefixed with the name of the provider in the context of the Elastic Agent.

For example, if a provider named foo provides {"key1": "value1", "key2": "value2"}, the key/value pairs are placed in {"foo" : {"key1": "value1", "key2": "value2"}}. To reference the keys, you would use {{foo.key1}} and {{foo.key2}}.

Provider configuration
edit

The provider configuration is specified under the top-level providers key in the elastic-agent.yml configuration. By default, all registered providers are enabled. If a provider cannot connect, it produces no mappings.

The following example shows two providers, local and local_dynamic, that supply custom keys:

providers:
  local:
    vars:
      foo: bar
  local_dynamic:
    vars:
      - item: key1
      - item: key2

To explicitly disable a provider, set enabled: false. Because all providers are prefixed and have no name collisions, the name of the provider is the key in the configuration.

providers:
  docker:
    enabled: false

Elastic Agent supports two broad types of providers: context and dynamic.

Context providers
edit

Context providers provide the current context of the running Elastic Agent, for example, agent information (id, version), host information (hostname, IP addresses), and environment information (environment variables).

They can only provide a single key/value mapping. Think of them as singletons; an update of a key/value mapping will result in a re-evaluation of the entire configuration. These providers are normally very static, but that’s not required. It is possible for a value to change resulting in re-evaluation.

Context providers use ECS naming when possible to ensure that documentation and understanding across projects is the same.

Elastic Agent supports the following context providers:

Local
edit

Provides custom keys to use as variables. For example:

providers:
  local:
    vars:
      foo: bar
Agent provider
edit

Provides information about the Elastic Agent. The available keys are:

Key Type Description

agent.id

string

Current agent ID

agent.version

object

Current agent version information object

agent.version.version

string

Current agent version

agent.version.commit

string

Version commit

agent.version.build_time

date

Version build time

agent.version.snapshot

boolean

Version is snapshot build

Host provider
edit

Provides information about the current host. The available keys are:

Key Type Description

host.name

string

Host name

host.platform

string

Host platform

host.architecture

string

Host architecture

host.ip[]

[]string

Host IP addresses

host.mac[]

[]string

Host MAC addresses

Env Provider
edit

Provides access to the environment variables as key/values.

For example, if you set the variable foo:

foo=bar elastic-agent run

You can reference the environment variable as ${env.foo}.

Dynamic Providers
edit

Dynamic providers provide an array of multiple key/value mappings. Each key/value mapping is combined with the previous context provider’s key/value mapping to provide a new unique key/value mapping that is used to generate a configuration.

Local dynamic provider
edit

Allows you to define multiple key/values to generate multiple configurations.

For example, the following agent policy defines a local dynamic provider that defines 3 values for item:

inputs:
 - type: logfile
   streams:
     - paths: "/var/${local_dynamic.my_var}/app.log"

providers:
  local_dynamic:
    items:
      - vars:
          my_var: key1
      - vars:
          my_var: key2
      - vars:
          my_var: key3

The configuration generated by this policy looks like:

inputs:
 - type: logfile
   streams:
     - paths: "/var/key1/app.log"
 - type: logfile
   streams:
     - paths: "/var/key2/app.log"
 - type: logfile
   streams:
   - paths: "/var/key3/app.log"
Docker Provider
edit

Provides inventory information from Docker. The available keys are:

Key Type Description

docker.id

string

ID of the container

docker.cmd

string

Arg path of container

docker.name

string

Name of the container

docker.image

string

Image of the container

docker.labels

string

Labels of the container

docker.ports

string

Ports of the container

docker.paths

object

Object of paths for the container

docker.paths.log

string

Log path of the container

Imagine that the Docker provider provides the following inventory:

[
    {
       "id": "1",
       "mapping:": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}},
       "processors": {"add_fields": {"container.name": "my-container"}}
    },
    {
        "id": "2",
        "mapping": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}},
        "processors": {"add_fields": {"container.name": "other-container"}}
    }
]

Elastic Agent automatically prefixes the result with docker:

---
[
    {"docker": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}}},
    {"docker": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}},
]
---

To set the log path dynamically in the configuration, use a variable in the Elastic Agent policy to return path information from the provider:

inputs:
  - type: logfile
    path: "${docker.paths.log}"

The policy generated by this configuration looks like:

inputs:
  - type: logfile
    path: "/var/log/containers/1.log"
    processors:
      - add_fields:
          container.name: my-container
  - type: logfile
    path: "/var/log/containers/2.log"
    processors:
      - add_fields:
          container.name: other-container

Conditions

edit

A condition is a boolean expression that you can specify in your agent policy to control whether a configuration is applied to the running Elastic Agent. You can set a condition on inputs, streams, or even processors.

In this example, the input is applied if the host platform is Linux:

inputs:
  - type: logfile
    streams:
      - paths:
         - /var/log/syslog
    condition: ${host.platform} == 'linux'

In this example, the stream is applied if the host platform is not Windows:

inputs:
  - type: system/metrics
    streams:
      - metricset: load
        data_stream.dataset: system.cpu
        condition: ${host.platform} != 'windows'

In this example, the processor is applied if the host platform is not Windows:

inputs:
  - type: system/metrics
    streams:
      - metricset: load
        data_stream.dataset: system.cpu
    processors:
      - add_fields:
          fields:
            platform: ${host.platform}
          to: host
        condition: ${host.platform} != 'windows'
Condition syntax
edit

The conditions supported by Elastic Agent are based on EQL's boolean syntax, but add support for variables from providers and functions to manipulate the values.

Supported operators:

  • Full PEMDAS math support for + - * / %.
  • Relational operators < <= >= > == !=
  • Logical operators and and or

Functions:

Types:

  • Booleans true and false
Condition examples
edit

Run only when a specific label is included.

arrayContains(${docker.labels}, 'monitor')

Skip on Linux platform or macOS.

${host.platform} != "linux" and ${host.platform} != "darwin"

Run only for specific labels.

arrayContains(${docker.labels}, 'monitor') or arrayContains(${docker.label}, 'production')

Function reference

edit

The condition syntax supports the following functions.

add
edit

add(Number, Number) Number

Usage:

add(1, 2) == 3
add(5, ${foo}) >= 5
arrayContains
edit

arrayContains(Array, String) Boolean

Usage:

arrayContains(${docker.labels}, 'monitor')
concat
edit

concat(String, String) String

Parameters are coerced into strings before the concatenation.

Usage:

concat("foo", "bar") == "foobar"
concat(${var1}, ${var2}) != "foobar"
divide
edit

divide(Number, Number) Number

Usage:

divide(25, 5) > 0
divide(${var1}, ${var2}) > 7
endsWith
edit

endsWith(String, String) Boolean

Usage:

endsWith("hello world", "hello") == true
endsWith(${var1}, "hello") != true
hasKey
edit

hasKey(Dictionary, String) Boolean

Usage:

hasKey(${host}, "platform")
indexOf
edit

indexOf(String, String, Number?) Number

Returns -1 if the string is not found.

Usage:

indexOf("hello", "llo") == 2
indexOf(${var1}, "hello") >= 0
length
edit

length(Array|Dictionary|string)

Usage:

length("foobar") > 2
length(${docker.labels}) > 0
length(${host}) > 2
match
edit

match(String, Regexp) boolean

Regexp supports Go’s regular expression syntax. Conditions that use regular expressions are more expensive to run. If speed is critical, consider using endWiths or startsWith.

Usage:

match("hello world", "^hello") == true
match(${var1}, "world$") == true
modulo
edit

modulo(number, number) Number

Usage:

modulo(25, 5) > 0
modulo(${var1}, ${var2}) == 0
multiply
edit

multiply(Number, Number) Number

Usage:

multiply(5, 5) == 25
multiple(${var1}, ${var2}) > x
number
edit

number(String) Integer

Usage:

number("42") == 42
number(${var1}) == 42
startsWith
edit

startsWith(String, String) Boolean

Usage:

startsWith("hello world", "hello") == true
startsWith(${var1}, "hello") != true
string
edit

string(Number) String

Usage:

string(42) == "42"
string(${var1}) == "42"
stringContains
edit

stringContains(String, String) Boolean

Usage:

stringContains("hello world", "hello") == true
stringContains(${var1}, "hello") != true
subtract
edit

subtract(Number, Number) Number

Usage:

subtract(5, 1) == 4
subtract(${foo}, 2) != 2

Debugging

edit

To debug configurations that include variable substitution and conditions, use the inspect command. This command shows the configuration that’s generated after variables are replaced and conditions are applied.

First run the Elastic Agent. For this example, we’ll use the following agent policy:

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    username: elastic
    password: changeme

providers:
  local_dynamic:
    items:
      - vars:
          key: value1
        processors:
          - add_fields:
              fields:
                custom: match1
              target: dynamic
      - vars:
          key: value2
        processors:
          - add_fields:
              fields:
                custom: match2
              target: dynamic
      - vars:
          key: value3
        processors:
          - add_fields:
              fields:
                custom: match3
              target: dynamic

inputs:
  - type: logfile
    enabled: true
    streams:
      - paths:
          - /var/log/{{local_dynamic.key}}

Then run elastic-agent inspect to see the generated configuration. For example:

$ ./elastic-agent inspect output -o default
[default] filebeat:
filebeat:
  inputs:
  - index: logs-generic-default
    paths:
    - /var/log/value1
    processors:
    - add_fields:
        fields:
          custom: match1
        target: dynamic
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    type: log
  - index: logs-generic-default
    paths:
    - /var/log/value2
    processors:
    - add_fields:
        fields:
          custom: match2
        target: dynamic
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    type: log
  - index: logs-generic-default
    paths:
    - /var/log/value3
    processors:
    - add_fields:
        fields:
          custom: match3
        target: dynamic
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    type: log
output:
  elasticsearch:
    hosts:
    - 127.0.0.1:9200
    password: changeme
    username: elastic

---
[default] FLEET_MONITORING:
output:
  elasticsearch:
    hosts:
    - 127.0.0.1:9200
    password: changeme
    type: elasticsearch
    username: elastic
programs:
- filebeat

---