Using Logstash with Shield
editUsing Logstash with Shield
editShield 1.0+ is compatible with Logstash 1.5 and above.
Logstash provides Elasticsearch output, input and filter plugins used to index and retrieve documents through HTTP, transport or client node protocols. All plugins support authentication and encryption over HTTP, while the output plugin additionally supports these features over the transport protocol.
When using the elasticsearch
output, only the transport
and http
protocol are supported (i.e. node
protocol is unsupported)
Creating a user
editBy default, the Shield plugin installs a dedicated user role that enables the creation of indices with names
that match the logstash-*
regular expression, along with privileges to read, scroll, index, update, and delete
documents on those indices:
logstash: cluster: indices:admin/template/get, indices:admin/template/put indices: 'logstash-*': indices:data/write/bulk, indices:data/write/delete, indices:data/write/update, indices:data/read/search, indices:data/read/scroll, create_index
See the Role Definition File section for information on modifying roles.
Create a user associated with the logstash
role on the Elasticsearch cluster, using the esusers
tool:
esusers useradd <username> -p <password> -r logstash
When using the transport protocol, the logstash user requires the predefined transport_client
role in addition to the logstash
role shown above (-r logstash,transport_client
).
Once you’ve created the user, you are ready to configure Logstash.
Connecting with HTTP/HTTPS
editWhen you set the protocol
option to http
, Logstash communicates with the Elasticsearch cluster through the REST APIs over HTTP.
Authentication for HTTP protocol
editHTTP protocol supports both basic auth and client-certificate authentication through the use of Public Key Infrastructure (PKI).
Basic Authentication
editThe input, filter, and output plugins all support HTTP Basic Authentication. To use basic authentication when connecting to an instance of Elasticsearch with Shield, you configure the plugins to include username and password credentials with each request. For example, the following snippet configures credentials for the output plugin. The credentials are configured the same way for each plugin type.
input { ... } output { elasticsearch { protocol => "http" ... user => ... # string password => ... # string } }
PKI Authentication
editElasticsearch Output version 1.0.1 onwards supports the use of X.509 client-certificate to authenticate Logstash requests. To enable this you need to set up the following configuration parameters:
input { ... } output { elasticsearch { protocol => "http" ... keystore => ... # string keystore_password => ... # string } }
SSL/TLS Configuration for HTTPS
editTo enable SSL/TLS encryption for HTTPS, use the following configuration block:
input { ... } output { elasticsearch { protocol => "http" ... ssl => true cacert => '/path/to/cert.pem' } }
Connecting with Transport protocol
editWhen you set the protocol
option to transport
, Logstash communicates with the Elasticsearch cluster through the same
protocol nodes use between each other. This avoids JSON un/marshalling and is therefore more efficient.
In order to unlock this option, it’s necessary to install an additional plugin in Logstash using the following command
to also get Shield compatibility within the transport
protocol
:
bin/plugin install logstash-output-elasticsearch_java bin/plugin install logstash-output-elasticsearch_java_shield
This is only necessary if you want to use the transport
protocol within Logstash. However, it is recommended
to use the default http
protocol, which means that you do not need to install any plugins, nor do you need to
use the elasticsearch_java
output.
Authentication for Transport protocol
editTransport protocol supports both basic auth and client-certificate authentication through the use of Public Key Infrastructure (PKI).
Basic Authentication
editTo connect to an instance of Elasticsearch with Shield using basic auth, set up the username and password credentials with the following configuration parameters:
input { ... } output { elasticsearch { protocol => "transport" ... user => ... # string password => ... # string } }
PKI Authentication
editTo connect to an instance of Elasticsearch with Shield using client-certificate authentication you need to setup the keystore path which contain the client’s certificate and the keystore password in the configuration:
input { ... } output { elasticsearch { protocol => "transport" ... ssl => true keystore => ... # string keystore_password => ... # string } }
SSL Configuration for Transport or Node protocols
editSpecify the paths to the keystore and truststore .jks
files with the following configuration parameters:
input { ... } output { elasticsearch { protocol => "transport" host => ... # string (optional) cluster => ... # string (optional) ... ssl => true keystore => ... # string keystore_password => ... # string truststore => ... # string truststore_password => ... # string } }
For more information on encryption and certificates, see the Securing Nodes section:
Failures
editLogstash raises an exception that halts the processing pipeline when the server’s certificate does not validate over SSL on any of the protocols discussed in this section. Same for the invalid user credentials.