Elastic Agent to Logstash for routing to multiple Elasticsearch clusters and additional destinations

edit

Elastic Agent to Logstash for routing to multiple Elasticsearch clusters and additional destinations

edit

Image showing Elastic Agent collecting and routing data to multiple destinations

Ingest model

Elastic Agent to Logstash to Elasticsearch clusters and/or additional destinations

Use when

Data collected by Elastic Agent needs to be routed to different Elasticsearch clusters or non-Elasticsearch destinations depending on the content

Example

Let’s take an example of a Windows workstation, for which we are collecting different types of logs using the System and Windows integrations. These logs need to be sent to different Elasticsearch clusters and to S3 for backup and a mechanism to send it to other destinations such as different SIEM solutions. In addition, the Elasticsearch destination is derived based on the type of datastream and an organization identifier.

In such use cases, agents send the data to Logstash as a routing mechanism to different destinations. Note that the System and Windows integrations must be installed on all Elasticsearch clusters to which the data is routed.

Sample config

input {
  elastic_agent {
    port => 5044
  }
}
filter {
  translate {
    source => "[http][host]"
    target => "[@metadata][tenant]"
    dictionary_path => "/etc/conf.d/logstash/tenants.yml"
  }
}
output {
  if [@metadata][tenant] == "tenant01" {
    elasticsearch {
      cloud_id => "<cloud id>"
      api_key => "<api key>"
    }
  } else if [@metadata][tenant] == "tenant02" {
    elasticsearch {
      cloud_id => "<cloud id>"
      api_key => "<api key>"
    }
  }
}

Resources

edit

Info on configuring Elastic Agent:

Info on Logstash and Logstash outputs:

Info on Elasticsearch:

Elasticsearch Guide

« Elastic Agent to Logstash to Elasticsearch: Logstash as a proxy Elastic Agent to proxy to Elasticsearch »