From version 5.0 onward, Shield is part of X-Pack. For more information, see Securing the Elastic Stack.

« Submitting Requests on Behalf of Other Users Securing Communications with Encryption and IP Filtering »

›

Auditing Security Events

Audit logs are disabled by default. To enable audit logging, set shield.audit.enabled in elasticsearch.yml:

shield.audit.enabled: true

You can enable auditing to keep track of security-related events such as authentication failures and refused connections. Logging these events enables you to monitor your cluster for suspicious activity and provides evidence in the event of an attack.

Shield provides two ways to persist audit logs:

A dedicated access.log file on the host’s file system.
An Elasticsearch index. The audit index can reside on the same cluster, or a separate cluster.

By default, audit events are written to the access.log. To facilitate browsing and analyzing the events, you can also enable indexing by setting shield.audit.outputs in elasticsearch.yml:

shield.audit.outputs: [index, logfile]

The index output type should be used in conjunction with the logfile output type. Because it is possible for the index output type to lose messages if the target index is unavailable, the logfile output type should be used as an official record of events.

Audit events are batched for indexing so there is a lag before events appear in the index. You can control how frequently batches of events are pushed to the index by setting shield.audit.index.flush_interval in elasticsearch.yml.

Log Entry Types

edit

Each audit event is represented by a single log entry. The type specified in a log entry represents the type of event that occurred:

anonymous_access_denied is logged when the request is denied due to missing authentication token.
authentication_failed is logged when the authentication token cannot be matched to a known user.
authentication_failed [<realm>] is logged for every realm that fails to present a valid authentication token. The value of <realm> is the realm type.
access_denied is logged when an authenticated user attempts an action the user does not have the privilege to perform.
access_granted is logged when an authenticated user attempts an action the user has the correct privilege to perform. In TRACE level all system (internal) actions are logged as well (in all other level they’re not logged to avoid cluttering of the logs.
tampered_request is logged when the request was detected to be tampered (typically relates to search/scroll requests when the scroll id is believed to be tampered)
connection_granted is logged when an incoming tcp connection has passed the ip filtering for a specific profile
connection_denied is logged when an incoming tcp connection did not pass the ip filtering for a specific profile

You can set the log level in logging.yml to control what types of events are logged. The following table lists the log entry types that are logged for each of the possible log levels.

Table 7. Log Entry Types and Levels

Log Level	Entry Type
`ERROR`	`authentication_failed`, `access_denied`, `tampered_request`, `connection_denied`
`WARN`	`authentication_failed`, `access_denied`, `tampered_request`, `connection_denied`, `anonymous_access_denied`
`INFO`	`authentication_failed`, `access_denied`, `tampered_request`, `connection_denied`, `anonymous_access_denied`, `access_granted`
`DEBUG`	(doesn’t output additional entry types beyond `INFO`, but extends the information emitted for each entry (see Log Entry Format below)
`TRACE`	`authentication_failed`, `access_denied`, `tampered_request`, `connection_denied`, `anonymous_access_denied`, `access_granted`, `connection_granted`, `authentication_failed [<realm>]`. In addition, internal system requests (self-management requests triggered by Elasticsearch itself) will also be logged for `access_granted` entry type.

For more information about configuring the audit logs, see Audit Log Settings.

Log Entry Format

edit

Every log entry contains the following information:

A timestamp that indicates when the event occurred.
The component/layer the event is associated with
The type of event that occurred.
Information about the event (varies depending to the type of event)

The format of a log entry is:

[<timestamp>] [<local_node_info>] [<layer>] [<entry_type>] <attribute_list>

Where:

<timestamp> - The time the event occurred. You can configure the timestamp format in logging.yml.
<local_node_info> - Information about the local node that this log entry is generated. You can control what node information is included by configuring the local node info settings.
<layer> - The layer from which this event originated: rest, transport or ip_filter.
<entry_type> - The type of event that occurred: anonymous_access_denied, authentication_failed, access_denied, access_granted, connection_granted, connection_denied.
<attribute_list> - A comma-separated list of key-value pairs that contain data pertaining to the event. Formatted as attr1=[val1], attr2=[val2],...). See Log Entry Attributes for the attributes that can be included for each type of event.

Table 8. Local Node Info Settings

Name	Default	Description
`shield.audit.logfile.prefix.emit_node_name`	true	Include or exclude the node’s name from the local node info.
`shield.audit.logfile.prefix.emit_node_host_address`	false	Include or exclude the node’s IP address from the local node info.
`shield.audit.logfile.prefix.emit_node_host_name`	false	Include or exclude the node’s host name from the local node info.

Log Entry Attributes

edit

The following tables show the attributes that can be associated with each type of event. The log level determines which attributes are included in a log entry.

Table 9. [rest] [anonymous_access_denied] attributes

Attribute	Minimum Log Level	Description
`origin_address`	WARN	The IP address from which the request originated
`uri`	WARN	The REST endpoint URI
`request_body`	DEBUG	The body of the request

Table 10. [rest] [authentication_failed] attributes

Attribute	Minimum Log Level	Description
`origin_address`	ERROR	The IP address from which the request originated
`principal`	ERROR	The principal (username) that failed authentication
`uri`	ERROR	The REST endpoint URI
`request_body`	DEBUG	The body of the request
`realm`	TRACE	The realm that failed to authenticate the user NOTE: A separate log entry is generated for each consulted realm.

Table 11. [transport] [anonymous_access_denied] attributes

Attribute	Minimum Log Level	Description
`origin_type`	WARN	Where the request originated: `rest` (request originated from a REST API request), `transport` (request was received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	WARN	The IP address from which the request originated
`action`	WARN	The name of the action that was executed
`request`	DEBUG	The type of request that was executed
`indices`	WARN	A comma-separated list of indices the request pertained to (when applicable)

Table 12. [transport] [authentication_failed] attributes

Attribute	Minimum Log Level	Description
`origin_type`	ERROR	Where the request originated: `rest` (request originated from a REST API request), `transport` (request was received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	ERROR	The IP address from which the request originated
`principal`	INFO	The principal (username) that failed authentication
`action`	ERROR	The name of the action that was executed
`request`	DEBUG	The type of request that was executed
`indices`	ERROR	A comma-separated list of indices the request pertained to (when applicable)
`realm`	TRACE	The realm that failed to authenticate the user NOTE: A separate log entry is generated for each consulted realm

Table 13. [transport] [access_granted] attributes

Attribute	Minimum Log Level	Description
`origin_type`	INFO	Where the request originated: `rest` (request originated from a REST API request), `transport` (request was received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	INFO	The IP address from which the request originated
`principal`	INFO	The principal (username) that failed authentication
`action`	INFO	The name of the action that was executed
`request`	DEBUG	The type of request that was executed
`indices`	INFO	A comma-separated list of indices the request pertained to (when applicable)

Table 14. [transport] [access_denied] attributes

Attribute	Minimum Log Level	Description
`origin_type`	ERROR	The type of the origin the request originated from. Can be either `rest` (request was originated from a rest API request), `transport` (request received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	ERROR	The IP address from which the request originated
`principal`	ERROR	The principal (username) that failed authentication
`action`	ERROR	The name of the action that was executed
`request`	DEBUG	The type of request that was executed
`indices`	ERROR	A comma-separated list of indices the request pertained to (when applicable)

Table 15. [transport] [tampered_request] attributes

Attribute	Minimum Log Level	Description
`origin_type`	ERROR	The type of the origin the request originated from. Can be either `rest` (request was originated from a rest API request), `transport` (request received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	ERROR	The IP address from which the request originated
`principal`	ERROR	TThe principal (username) that failed authentication
`action`	ERROR	The name of the action that was executed
`request`	DEBUG	The type of request that was executed
`indices`	ERROR	A comma-separated list of indices the request pertained to (when applicable)

Table 16. [ip_filter] [connection_granted] attributes

Attribute	Minimum Log Level	Description
`origin_address`	TRACE	The IP address from which the request originated
`transport_profile`	TRACE	The principal (username) that failed authentication
`rule`	TRACE	The IP filtering rule that granted the request

Table 17. [ip_filter] [connection_denied] attributes

Attribute	Minimum Log Level	Description
`origin_address`	ERROR	The IP address from which the request originated
`transport_profile`	ERROR	The principal (username) that failed authentication
`rule`	ERROR	The IP filtering rule that denied the request

Audit Log Settings

edit

You configure audit log options in the logging.yml file located in CONFIG_DIR/shield. The following snippet shows the default log configuration:

Default logging.yml File.

logger:
  shield.audit.logfile: INFO, access_log

additivity:
  shield.audit.logfile: false

appender:

  access_log:
    type: dailyRollingFile
    file: ${path.logs}/${cluster.name}-access.log
    datePattern: "'.'yyyy-MM-dd"
    layout:
      type: pattern
      conversionPattern: "[%d{ISO8601}] %m%n"

By default, audit information is appended to the access.log file located in the standard Elasticsearch logs directory (typically located at $ES_HOME/logs).

Storing Audit Logs in an Elasticsearch Index

edit

In addition to logging to a file, you can store audit logs in Elasticsearch indices. You can configure the following settings in elasticsearch.yml to control how audit log entries are indexed.

Table 18. Audit Log Index Settings

Attribute	Default Setting	Description
`shield.audit.outputs`	`logfile`	Must be set to `index` or `[index, logfile]` to enable audit logging.
`shield.audit.index.bulk_size`	`1000`	Controls how many audit events are batched into a single write
`shield.audit.index.flush_interval`	`1s`	Controls how often to flush buffered events into the index
`shield.audit.index.rollover`	`daily`	Controls how often to roll over to a new index: `hourly`, `daily`, `weekly`, `monthly`
`shield.audit.index.events.include`	`anonymous_access_denied, authentication_failed, access_granted, access_denied, tampered_request, connection_granted, connection_denied`	The audit events to be indexed. Valid values: `anonymous_access_denied, authentication_failed, access_granted, access_denied, tampered_request, connection_granted, connection_denied`, `system_access_granted`. `_all` is a special value that includes all types.
`shield.audit.index.events.exclude`	`system_access_granted`	The audit events to exclude from indexing. By default, `system_access_granted` events are excluded. If you include these events, all internal node communications are indexed, which greatly increases the size of the audit index.

You can also configure standard index settings for the audit log indices in shield.audit.index.settings. For example, the following configuration sets the number of shards and replicas to 1 for the audit indices:

shield.audit.index.settings:
  index:
    number_of_shards: 1
    number_of_replicas: 1

Forwarding Audit Logs to a Remote Cluster

edit

To index audit events to a remote Elasticsearch cluster, you configure the following shield.audit.index.client settings.

Table 19. Remote Audit Log Indexing Settings

Attribute	Default	Description
`shield.audit.index.client.hosts`	None	Comma separated list of host:port pairs. These hosts should be nodes in the cluster to which you want to index.
`shield.audit.index.client.cluster.name`	None	The name of the remote cluster.
`shield.audit.index.client.shield.user`	None	The username:password pair used to authenticate with the remote cluster.

You can pass additional settings to the remote client using the shield.audit.index.client namespace. For example, to allow the remote client to discover all of the nodes in the remote cluster you could set client.transport.sniff:

shield.audit.index.client.transport.sniff: true

« Submitting Requests on Behalf of Other Users Securing Communications with Encryption and IP Filtering »