elasticsearch

When a healthcheck_path config is provided, this additional flag can be used to specify whether the healthcheck_path is appended to the existing path (default) or is treated as the absolute URL path.

For example, if hosts url is "http://localhost:9200/es" and healthcheck_path is "/health", the health check url will be:

Protocol agnostic (i.e. non-http, non-java specific) configs go here Protocol agnostic methods The Elasticsearch action to perform. Valid actions are:

For more details on actions, check out the Elasticsearch bulk API documentation

The .cer or .pem file to validate the server’s certificate

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

Enable doc_as_upsert for update mode. Create a new document with source if document_id doesn’t exist in Elasticsearch

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

The document type to write events to. Generally you should try to write only similar events to the same type. String expansion %{foo} works here. Unless you set document_type, the event type will be used if it exists otherwise the document type will be assigned the value of logs

Disable or enable metric logging for this specific plugin instance by default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

Set the Elasticsearch errors in the whitelist that you don’t want to log. A useful example is when you want to skip all 409 errors which are document_already_exists_exception.

This plugin uses the bulk index API for improved indexing performance. This setting defines the maximum sized bulk request Logstash will make. You may want to increase this to be in line with your pipeline’s batch size. If you specify a number larger than the batch size of your pipeline it will have no effect, save for the case where a filter increases the size of an inflight batch by outputting events.

When a backend is marked down a HEAD request will be sent to this path in the background to see if it has come back again before it is once again eligible to service requests. If you have custom firewall rules you may need to change this NOTE: any query parameters present in the URL or query_params config option will be removed

Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts parameter. Remember the http protocol uses the http address (eg. 9200, not 9300). "127.0.0.1" ["127.0.0.1:9200","127.0.0.2:9200"] ["http://127.0.0.1"] ["https://127.0.0.1:9200"] ["https://127.0.0.1:9200/mypath"] (If using a proxy on a subpath) It is important to exclude dedicated master nodes from the hosts list to prevent LS from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes in Elasticsearch.

Any special characters present in the URLs here MUST be URL escaped! This means # should be put in as %23 for instance.

Add a unique ID to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type, for example, if you have 2 grok filters. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.

output {
 stdout {
   id => "my_plugin_id"
 }
}

The amount of time since last flush before a flush is forced.

This setting helps ensure slow event rates don’t get stuck in Logstash. For example, if your flush_size is 100, and you have received 10 events, and it has been more than idle_flush_time seconds since the last flush, Logstash will flush those 10 events automatically.

This helps keep both fast and slow log streams moving along in near-real-time.

The index to write events to. This can be dynamic using the %{foo} syntax. The default value will partition your indices by day so you can more easily delete old data or only search specific date ranges. Indexes may not contain uppercase characters. For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}. LS uses Joda to format the index pattern from event timestamp. Joda formats are defined here.

The keystore used to present a certificate to the server. It can be either .jks or .p12

Set the truststore password

From Logstash 1.3 onwards, a template is applied to Elasticsearch during Logstash’s startup if one with the name template_name does not already exist. By default, the contents of this template is the default template for logstash-%{+YYYY.MM.dd} which always matches indices based on the pattern logstash-*. Should you require support for other index names, or would like to change the mappings in the template in general, a custom template can be specified by setting template to the path of a template file.

Setting manage_template to false disables this feature. If you require more control over template creation, (e.g. creating indices dynamically based on field names) you should set manage_template to false and use the REST API to apply your templates manually.

Pass a set of key value pairs as the URL query string. This query string is added to every host listed in the hosts configuration. If the hosts list contains urls that already have query strings, the one specified here will be appended.

For child documents, ID of the associated parent. This can be dynamic using the %{foo} syntax.

Password to authenticate to a secure Elasticsearch cluster

HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps the root path for the Elasticsearch HTTP API lives. Note that if you use paths as components of URLs in the hosts field you may not also set this field. That will raise an error at startup

Set which ingest pipeline you wish to execute for an event. You can also use event dependent configuration here like pipeline => "%{INGEST_PIPELINE}"

While the output tries to reuse connections efficiently we have a maximum. This sets the maximum number of open connections the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

While the output tries to reuse connections efficiently we have a maximum per endpoint. This sets the maximum number of open connections per endpoint the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

Set the address of a forward HTTP proxy. This used to accept hashes as arguments but now only accepts arguments of the URI type to prevent leaking credentials.

How frequently, in seconds, to wait between resurrection attempts. Resurrection is the process by which backend endpoints marked down are checked to see if they have come back to life

Set initial interval in seconds between bulk retries. Doubled on each retry up to retry_max_interval

Set max interval in seconds between bulk retries.

The number of times Elasticsearch should internally retry an update/upserted document See the partial updates for more info

A routing override to be applied to all processed events. This can be dynamic using the %{foo} syntax.

Set script name for scripted update mode

Set the language of the used script. If not set, this defaults to painless in ES 5.0

Define the type of script referenced by "script" variable inline : "script" contains inline script indexed : "script" contains the name of script directly indexed in elasticsearch file : "script" contains the name of script stored in elasticseach’s config directory

Set variable name passed to script (scripted update)

if enabled, script is in charge of creating non-existent document (scripted update)

This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list. Note: This will return ALL nodes with HTTP enabled (including master nodes!). If you use this with master nodes, you probably want to disable HTTP on them by setting http.enabled to false in their elasticsearch.yml. You can either use the sniffing option or manually enter multiple Elasticsearch hosts using the hosts parameter.

How long to wait, in seconds, between sniffing attempts

Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme is specified in the URLs listed in hosts. If no explicit protocol is specified plain HTTP will be used. If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in hosts

Option to validate the server’s certificate. Disabling this severely compromises security. For more information on disabling certificate verification please read https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf

You can set the path to your own template here, if you so desire. If not set, the included template will be used.

This configuration option defines how the template is named inside Elasticsearch. Note that if you have used the template management features and subsequently change this, you will need to prune the old template manually, e.g.

curl -XDELETE <http://localhost:9200/_template/OldTemplateName?pretty>

where OldTemplateName is whatever the former setting was.

The template_overwrite option will always overwrite the indicated template in Elasticsearch with either the one indicated by template or the included one. This option is set to false by default. If you always want to stay up to date with the template provided by Logstash, this option could be very useful to you. Likewise, if you have your own template file managed by puppet, for example, and you wanted to be able to update it regularly, this option could help there as well.

Please note that if you are using your own customized version of the Logstash template (logstash), setting this to true will make Logstash to overwrite the "logstash" template (i.e. removing all customized settings)

Set the timeout, in seconds, for network operations and requests sent Elasticsearch. If a timeout occurs, the request will be retried.

The JKS truststore to validate the server’s certificate. Use either :truststore or :cacert

Set the truststore password

Set upsert content for update mode.s Create a new document with this parameter as json string if document_id doesn’t exists

Username to authenticate to a secure Elasticsearch cluster

How long to wait before checking if the connection is stale before executing a request on a connection using keepalive. You may want to set this lower, if you get connection errors regularly Quoting the Apache commons docs (this client is based Apache Commmons): Defines period of inactivity in milliseconds after which persistent connections must be re-validated prior to being leased to the consumer. Non-positive value passed to this method disables connection validation. This check helps detect connections that have become stale (half-closed) while kept inactive in the pool. See these docs for more info

The version to use for indexing. Use sprintf syntax like %{my_version} to use a field value here. See https://www.elastic.co/blog/elasticsearch-versioning-support.

The version_type to use for indexing. See https://www.elastic.co/blog/elasticsearch-versioning-support. See also https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#_version_types

when we no longer support the :legacy type This is hacky, but it can only be herne