Parse data using an ingest pipeline

When you use Elasticsearch for output, you can configure Winlogbeat to use an ingest pipeline to pre-process documents before the actual indexing takes place in Elasticsearch. An ingest pipeline is a convenient processing option when you want to do some extra processing on your data, but you do not require the full power of Logstash. For example, you can create an ingest pipeline in Elasticsearch that consists of one processor that removes a field in a document followed by another processor that renames a field.

After defining the pipeline in Elasticsearch, you simply configure Winlogbeat to use the pipeline. To configure Winlogbeat, you specify the pipeline ID in the parameters option under elasticsearch in the winlogbeat.yml file:

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: my_pipeline_id

For example, let’s say that you’ve defined the following pipeline in a file named pipeline.json:

{
    "description": "Test pipeline",
    "processors": [
        {
            "lowercase": {
                "field": "agent.name"
            }
        }
    ]
}

To add the pipeline in Elasticsearch, you would run:

curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_ingest/pipeline/test-pipeline' [email protected]

Then in the winlogbeat.yml file, you would specify:

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: "test-pipeline"

When you run Winlogbeat, the value of agent.name is converted to lowercase before indexing.

For more information about defining a pre-processing pipeline, see the ingest pipeline documentation.