elasticsearch

edit

Compatibility Note

Starting with Elasticsearch 5.3, there’s an HTTP setting called http.content_type.required. If this option is set to true, and you are using Logstash 2.4 through 5.2, you need to update the Elasticsearch output plugin to version 6.2.5 or higher.

This plugin is the recommended method of storing logs in Elasticsearch. If you plan on using the Kibana web interface, you’ll want to use this output.

This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0. We strongly encourage the use of HTTP over the node protocol for a number of reasons. HTTP is only marginally slower, yet far easier to administer and work with. When using the HTTP protocol one may upgrade Elasticsearch versions without having to upgrade Logstash in lock-step.

You can learn more about Elasticsearch at https://www.elastic.co/products/elasticsearch

Template management for Elasticsearch 5.x

edit

Index template for this version (Logstash 5.0) has been changed to reflect Elasticsearch’s mapping changes in version 5.0. Most importantly, the subfield for string multi-fields has changed from .raw to .keyword to match ES default behavior.

  • Users installing ES 5.x and LS 5.x ** This change will not affect you and you will continue to use the ES defaults.
  • Users upgrading from LS 2.x to LS 5.x with ES 5.x ** LS will not force upgrade the template, if logstash template already exists. This means you will still use .raw for sub-fields coming from 2.x. If you choose to use the new template, you will have to reindex your data after the new template is installed.

Retry Policy

edit

The retry policy has changed significantly in the 2.2.0 release. This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience either partial or total failures.

The following errors are retried infinitely:

  • Network errors (inability to connect)
  • 429 (Too many requests) and
  • 503 (Service unavailable) errors

409 exceptions are no longer retried. Please set a higher retry_on_conflict value if you experience 409 exceptions. It is more performant for Elasticsearch to retry these exceptions than this plugin.

DNS Caching

edit

This plugin uses the JVM to lookup DNS entries and is subject to the value of networkaddress.cache.ttl, a global setting for the JVM.

As an example, to set your DNS TTL to 1 second you would set the LS_JAVA_OPTS environment variable to -Dnetworkaddress.cache.ttl=1.

Keep in mind that a connection with keepalive enabled will not reevaluate its DNS value while the keepalive is in effect.

 

Synopsis

edit

This plugin supports the following configuration options:

Required configuration options:

elasticsearch {
}

Available configuration options:

Setting Input type Required Default value

action

string

No

"index"

cacert

a valid filesystem path

No

codec

codec

No

"plain"

doc_as_upsert

boolean

No

false

document_id

string

No

document_type

string

No

failure_type_logging_whitelist

array

No

[]

flush_size

number

No

500

healthcheck_path

string

No

"/"

hosts

array

No

["127.0.0.1"]

idle_flush_time

number

No

1

index

string

No

"logstash-%{+YYYY.MM.dd}"

keystore

a valid filesystem path

No

keystore_password

password

No

manage_template

boolean

No

true

parent

string

No

nil

password

password

No

path

string

No

pipeline

string

No

nil

pool_max

number

No

1000

pool_max_per_route

number

No

100

proxy

<<,>>

No

resurrect_delay

number

No

5

retry_initial_interval

number

No

2

retry_max_interval

number

No

64

retry_on_conflict

number

No

1

routing

string

No

script

string

No

""

script_lang

string

No

"painless"

script_type

string, one of ["inline", "indexed", "file"]

No

["inline"]

script_var_name

string

No

"event"

scripted_upsert

boolean

No

false

sniffing

boolean

No

false

sniffing_delay

number

No

5

ssl

boolean

No

ssl_certificate_verification

boolean

No

true

template

a valid filesystem path

No

template_name

string

No

"logstash"

template_overwrite

boolean

No

false

timeout

number

No

60

truststore

a valid filesystem path

No

truststore_password

password

No

upsert

string

No

""

user

string

No

validate_after_inactivity

number

No

10000

workers

<<,>>

No

1

Details

edit

 

action

edit
  • Value type is string
  • Default value is "index"

Protocol agnostic (i.e. non-http, non-java specific) configs go here Protocol agnostic methods The Elasticsearch action to perform. Valid actions are:

  • index: indexes a document (an event from Logstash).
  • delete: deletes a document by id (An id is required for this action)
  • create: indexes a document, fails if a document by that id already exists in the index.
  • update: updates a document by id. Update has a special case where you can upsert — update a document if not already present. See the upsert option. NOTE: This does not work and is not supported in Elasticsearch 1.x. Please upgrade to ES 2.x or greater to use this feature with Logstash!
  • A sprintf style string to change the action based on the content of the event. The value %{[foo]} would use the foo field for the action

For more details on actions, check out the Elasticsearch bulk API documentation

cacert

edit
  • Value type is path
  • There is no default value for this setting.

The .cer or .pem file to validate the server’s certificate

codec

edit
  • Value type is codec
  • Default value is "plain"

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

doc_as_upsert

edit
  • Value type is boolean
  • Default value is false

Enable doc_as_upsert for update mode. Create a new document with source if document_id doesn’t exist in Elasticsearch

document_id

edit
  • Value type is string
  • There is no default value for this setting.

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

document_type

edit
  • Value type is string
  • There is no default value for this setting.

The document type to write events to. Generally you should try to write only similar events to the same type. String expansion %{foo} works here. Unless you set document_type, the event type will be used if it exists otherwise the document type will be assigned the value of logs

failure_type_logging_whitelist

edit
  • Value type is array
  • Default value is []

Set the Elasticsearch errors in the whitelist that you don’t want to log. A useful example is when you want to skip all 409 errors which are document_already_exists_exception.

flush_size

edit
  • Value type is number
  • Default value is 500

This plugin uses the bulk index API for improved indexing performance. This setting defines the maximum sized bulk request Logstash will make You you may want to increase this to be in line with your pipeline’s batch size. If you specify a number larger than the batch size of your pipeline it will have no effect, save for the case where a filter increases the size of an inflight batch by outputting events.

healthcheck_path

edit
  • Value type is string
  • Default value is "/"

When a backend is marked down a HEAD request will be sent to this path in the background to see if it has come back again before it is once again eligible to service requests. If you have custom firewall rules you may need to change this

hosts

edit
  • Value type is array
  • Default value is ["127.0.0.1"]

Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts parameter. Remember the http protocol uses the http address (eg. 9200, not 9300). "127.0.0.1" ["127.0.0.1:9200","127.0.0.2:9200"] ["http://127.0.0.1"] ["https://127.0.0.1:9200"] ["https://127.0.0.1:9200/mypath"] (If using a proxy on a subpath) It is important to exclude dedicated master nodes from the hosts list to prevent LS from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes in Elasticsearch.

idle_flush_time

edit
  • Value type is number
  • Default value is 1

The amount of time since last flush before a flush is forced.

This setting helps ensure slow event rates don’t get stuck in Logstash. For example, if your flush_size is 100, and you have received 10 events, and it has been more than idle_flush_time seconds since the last flush, Logstash will flush those 10 events automatically.

This helps keep both fast and slow log streams moving along in near-real-time.

index

edit
  • Value type is string
  • Default value is "logstash-%{+YYYY.MM.dd}"

The index to write events to. This can be dynamic using the %{foo} syntax. The default value will partition your indices by day so you can more easily delete old data or only search specific date ranges. Indexes may not contain uppercase characters. For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}. LS uses Joda to format the index pattern from event timestamp. Joda formats are defined here.

keystore

edit
  • Value type is path
  • There is no default value for this setting.

The keystore used to present a certificate to the server. It can be either .jks or .p12

keystore_password

edit
  • Value type is password
  • There is no default value for this setting.

Set the truststore password

manage_template

edit
  • Value type is boolean
  • Default value is true

Starting in Logstash 1.3 (unless you set option manage_template to false) a default mapping template for Elasticsearch will be applied, if you do not already have one set to match the index pattern defined (default of logstash-%{+YYYY.MM.dd}), minus any variables. For example, in this case the template will be applied to all indices starting with logstash-*

If you have dynamic templating (e.g. creating indices based on field names) then you should set manage_template to false and use the REST API to upload your templates manually.

parent

edit
  • Value type is string
  • Default value is nil

For child documents, ID of the associated parent. This can be dynamic using the %{foo} syntax.

password

edit
  • Value type is password
  • There is no default value for this setting.

Password to authenticate to a secure Elasticsearch cluster

path

edit
  • Value type is string
  • There is no default value for this setting.

HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps the root path for the Elasticsearch HTTP API lives. Note that if you use paths as components of URLs in the hosts field you may not also set this field. That will raise an error at startup

pipeline

edit
  • Value type is string
  • Default value is nil

Set which ingest pipeline you wish to execute for an event

pool_max

edit
  • Value type is number
  • Default value is 1000

While the output tries to reuse connections efficiently we have a maximum. This sets the maximum number of open connections the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

pool_max_per_route

edit
  • Value type is number
  • Default value is 100

While the output tries to reuse connections efficiently we have a maximum per endpoint. This sets the maximum number of open connections per endpoint the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

proxy

edit
  • Value type is string
  • There is no default value for this setting.

Set the address of a forward HTTP proxy. Can be either a string, such as http://localhost:123 or a hash in the form of {host: 'proxy.org' port: 80 scheme: 'http'}. Note, this is NOT a SOCKS proxy, but a plain HTTP proxy

resurrect_delay

edit
  • Value type is number
  • Default value is 5

How frequently, in seconds, to wait between resurrection attempts. Resurrection is the process by which backend endpoints marked down are checked to see if they have come back to life

retry_initial_interval

edit
  • Value type is number
  • Default value is 2

Set initial interval in seconds between bulk retries. Doubled on each retry up to retry_max_interval

retry_max_interval

edit
  • Value type is number
  • Default value is 64

Set max interval in seconds between bulk retries.

retry_on_conflict

edit
  • Value type is number
  • Default value is 1

The number of times Elasticsearch should internally retry an update/upserted document See the partial updates for more info

routing

edit
  • Value type is string
  • There is no default value for this setting.

A routing override to be applied to all processed events. This can be dynamic using the %{foo} syntax.

script

edit
  • Value type is string
  • Default value is ""

Set script name for scripted update mode

script_lang

edit
  • Value type is string
  • Default value is "painless"

Set the language of the used script. If not set, this defaults to painless in ES 5.0

script_type

edit
  • Value can be any of: inline, indexed, file
  • Default value is ["inline"]

Define the type of script referenced by "script" variable inline : "script" contains inline script indexed : "script" contains the name of script directly indexed in elasticsearch file : "script" contains the name of script stored in elasticseach’s config directory

script_var_name

edit
  • Value type is string
  • Default value is "event"

Set variable name passed to script (scripted update)

scripted_upsert

edit
  • Value type is boolean
  • Default value is false

if enabled, script is in charge of creating non-existent document (scripted update)

sniffing

edit
  • Value type is boolean
  • Default value is false

This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list. Note: This will return ALL nodes with HTTP enabled (including master nodes!). If you use this with master nodes, you probably want to disable HTTP on them by setting http.enabled to false in their elasticsearch.yml. You can either use the sniffing option or manually enter multiple Elasticsearch hosts using the hosts parameter.

sniffing_delay

edit
  • Value type is number
  • Default value is 5

How long to wait, in seconds, between sniffing attempts

ssl

edit
  • Value type is boolean
  • There is no default value for this setting.

Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme is specified in the URLs listed in hosts. If no explicit protocol is specified plain HTTP will be used. If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in hosts

ssl_certificate_verification

edit
  • Value type is boolean
  • Default value is true

Option to validate the server’s certificate. Disabling this severely compromises security. For more information on disabling certificate verification please read https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf

template

edit
  • Value type is path
  • There is no default value for this setting.

You can set the path to your own template here, if you so desire. If not set, the included template will be used.

template_name

edit
  • Value type is string
  • Default value is "logstash"

This configuration option defines how the template is named inside Elasticsearch. Note that if you have used the template management features and subsequently change this, you will need to prune the old template manually, e.g.

curl -XDELETE <http://localhost:9200/_template/OldTemplateName?pretty>

where OldTemplateName is whatever the former setting was.

template_overwrite

edit
  • Value type is boolean
  • Default value is false

The template_overwrite option will always overwrite the indicated template in Elasticsearch with either the one indicated by template or the included one. This option is set to false by default. If you always want to stay up to date with the template provided by Logstash, this option could be very useful to you. Likewise, if you have your own template file managed by puppet, for example, and you wanted to be able to update it regularly, this option could help there as well.

Please note that if you are using your own customized version of the Logstash template (logstash), setting this to true will make Logstash to overwrite the "logstash" template (i.e. removing all customized settings)

timeout

edit
  • Value type is number
  • Default value is 60

Set the timeout, in seconds, for network operations and requests sent Elasticsearch. If a timeout occurs, the request will be retried.

truststore

edit
  • Value type is path
  • There is no default value for this setting.

The JKS truststore to validate the server’s certificate. Use either :truststore or :cacert

truststore_password

edit
  • Value type is password
  • There is no default value for this setting.

Set the truststore password

upsert

edit
  • Value type is string
  • Default value is ""

Set upsert content for update mode.s Create a new document with this parameter as json string if document_id doesn’t exists

user

edit
  • Value type is string
  • There is no default value for this setting.

Username to authenticate to a secure Elasticsearch cluster

validate_after_inactivity

edit
  • Value type is number
  • Default value is 10000

How long to wait before checking if the connection is stale before executing a request on a connection using keepalive. You may want to set this lower, if you get connection errors regularly Quoting the Apache commons docs (this client is based Apache Commmons): Defines period of inactivity in milliseconds after which persistent connections must be re-validated prior to being leased to the consumer. Non-positive value passed to this method disables connection validation. This check helps detect connections that have become stale (half-closed) while kept inactive in the pool. See these docs for more info

workers

edit
  • Value type is string
  • Default value is 1

Output workers are no longer supported. Please use plugin workers