Auditing

Audit logs are disabled by default. To enable this functionality the following setting should be added to the elasticsearch.yml file:

shield.audit.enabled: true

The audit functionality was added to keep track of important events occurring in elasticsearch, primarily around security concerns. Keeping track and persisting these events is essential for any secured environment and potentially provides evidence for suspicious/malicious activity on the elasticsearch cluster.

Currently, the only output for these events are dedicated access.log files stored on the host’s file system.

Log Entry Types

edit

Each audit related event that occurs is represented by a single log entry of a specific type (the type represents the type of the event that occurred). Here are the possible log entry types:

anonymous_access_denied is logged when the request is denied due to missing authentication token.
authentication_failed is logged when the authentication token cannot be matched to a known user.
authentication_failed [<realm>] is logged for every realm that fails to present a valid authentication token. The value of <realm> is the realm type.
access_denied is logged when an authenticated user attempts an action the user does not have the privilege to perform.
access_granted is logged when an authenticated user attempts an action the user has the correct privilege to perform. In TRACE level all system (internal) actions are logged as well (in all other level they’re not logged to avoid cluttering of the logs.
tampered_request is logged when the request was detected to be tampered (typically relates to search/scroll requests when the scroll id is believed to be tampered)
connection_granted is logged when an incoming tcp connection has passed the ip filtering for a specific profile
connection_denied is logged when an incoming tcp connection did not pass the ip filtering for a specific profile

To avoid needless proliferation of log entries, Shield enables you to control what entry types should be logged. This can be done by setting the logging level. The following table lists the log entry types that will be logged for each of the possible log levels:

Table 9. Log Entry Types and Levels

Log Level	Entry Type
`ERROR`	`authentication_failed`, `access_denied`, `tampered_request`, `connection_denied`
`WARN`	`authentication_failed`, `access_denied`, `tampered_request`, `connection_denied`, `anonymous_access_denied`
`INFO`	`authentication_failed`, `access_denied`, `tampered_request`, `connection_denied`, `anonymous_access_denied`, `access_granted`
`DEBUG`	(doesn’t output additional entry types beyond `INFO`, but extends the information emitted for each entry (see Log Entry Format below)
`TRACE`	`authentication_failed`, `access_denied`, `tampered_request`, `connection_denied`, `anonymous_access_denied`, `access_granted`, `connection_granted`, `authentication_failed [<realm>]`. In addition, internal system requests (self-management requests triggered by elasticsearch itself) will also be logged for `access_granted` entry type.

Log Entry Format

edit

As mentioned above, every log entry represents an event that occurred in the system. As such, each entry is associated with a timestamp (at which the event occurred), the component/layer the event is associated with and the entry/event type. In addition, every log entry (depending ot its type) carries addition information about the event.

The format of a log entry is shown below:

[<timestamp>] [<local_node_info] [<layer>] [<entry_type>] <attribute_list>

Where:

<timestamp> - the timestamp of the entries (in the fomrat configured in logging.yml as shown above)
<local_node_info - additional information about the local node that this log entry is printed from (the table below shows how this information can be controlled via settings)
<layer> - the layer from which this entry relates to. Can be either rest, transport or ip_filter
<entry_type> - the type of the entry as discussed above. Can be either anonymous_access_denied, authentication_failed, access_denied, access_granted, connection_granted, connection_denied.
<attribute_list> - A comma-separated list of attribute carrying data relevant to the occurred event (formatted as attr1=[val1], attr2=[val2],...)

Table 10. Local Node Info Settings

Name	Default	Description
`shield.audit.logfile.prefix.emit_node_name`	true	When set to `true`, the local node’s name will be emitted
`shield.audit.logfile.prefix.emit_node_host_address`	false	When set to `true`, the local node’s IP address will be emitted
`shield.audit.logfile.prefix.emit_node_host_name`	false	When set to `true`, the local node’s host name will be emitted

The following tables describe the possible attributes each entry type can carry (the attributes that will be available depend on the configured log level):

Table 11. [rest] [anonymous_access_denied] attributes

Attribute	Minimum Log Level	Description
`origin_address`	WARN	The address the rest request origins from
`uri`	WARN	The REST endpoint URI
`request_body`	DEBUG	The body of the request

Table 12. [rest] [authentication_failed] attributes

Attribute	Minimum Log Level	Description
`origin_address`	ERROR	The address the rest request origins from
`principal`	ERROR	The principal (username) that failed to authenticate
`uri`	ERROR	The REST endpoint URI
`request_body`	DEBUG	The body of the request
`realm`	TRACE	The realm that failed to authenticate the user. NOTE: A separate entry will be printed for each of the consulted realms

Table 13. [transport] [anonymous_access_denied] attributes

Attribute	Minimum Log Level	Description
`origin_type`	WARN	The type of the origin the request originated from. Can be either `rest` (request was originated from a rest API request), `transport` (request received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	WARN	The address the request origins from
`action`	WARN	The name of the action that was executed
`request`	DEBUG	The type of the request that was executed
`indices`	WARN	A comma-separated list of indices this request relates to (when applicable)

Table 14. [transport] [authentication_failed] attributes

Attribute	Minimum Log Level	Description
`origin_type`	ERROR	The type of the origin the request originated from. Can be either `rest` (request was originated from a rest API request), `transport` (request received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	ERROR	The address the request origins from
`principal`	ERROR	The principal (username) that failed to authenticate
`action`	ERROR	The name of the action that was executed
`request`	DEBUG	The type of the request that was executed
`indices`	ERROR	A comma-separated list of indices this request relates to (when applicable)
`realm`	TRACE	The realm that failed to authenticate the user. NOTE: A separate entry will be printed for each of the consulted realms

Table 15. [transport] [access_granted] attributes

Attribute	Minimum Log Level	Description
`origin_type`	INFO	The type of the origin the request originated from. Can be either `rest` (request was originated from a rest API request), `transport` (request received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	INFO	The address the request origins from
`principal`	INFO	The principal (username) that failed to authenticate
`action`	INFO	The name of the action that was executed
`request`	DEBUG	The type of the request that was executed
`indices`	INFO	A comma-separated list of indices this request relates to (when applicable)

Table 16. [transport] [access_denied] attributes

Attribute	Minimum Log Level	Description
`origin_type`	ERROR	The type of the origin the request originated from. Can be either `rest` (request was originated from a rest API request), `transport` (request received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	ERROR	The address the request origins from
`principal`	ERROR	The principal (username) that failed to authenticate
`action`	ERROR	The name of the action that was executed
`request`	DEBUG	The type of the request that was executed
`indices`	ERROR	A comma-separated list of indices this request relates to (when applicable)

Table 17. [transport] [tampered_request] attributes

Attribute	Minimum Log Level	Description
`origin_type`	ERROR	The type of the origin the request originated from. Can be either `rest` (request was originated from a rest API request), `transport` (request received on the transport channel), `local_node` (the local node issued the request)
`origin_address`	ERROR	The address the request origins from
`principal`	ERROR	The principal (username) that failed to authenticate
`action`	ERROR	The name of the action that was executed
`request`	DEBUG	The type of the request that was executed
`indices`	ERROR	A comma-separated list of indices this request relates to (when applicable)

Table 18. [ip_filter] [connection_granted] attributes

Attribute	Minimum Log Level	Description
`origin_address`	TRACE	The address the request origins from
`transport_profile`	TRACE	The principal (username) that failed to authenticate
`rule`	TRACE	The IP filtering rule that granted the request

Table 19. [ip_filter] [connection_denied] attributes

Attribute	Minimum Log Level	Description
`origin_address`	ERROR	The address the request origins from
`transport_profile`	ERROR	The principal (username) that failed to authenticate
`rule`	ERROR	The IP filtering rule that denied the request

Audit Logs Settings

edit

As mentioned above, the audit logs are configured in the logging.yml file located in Shield’s config directory. The following snippet shows the default logging configuration:

Default logging.yml File.

logger:
  shield.audit.logfile: INFO, access_log

additivity:
  shield.audit.logfile: false

appender:

  access_log:
    type: dailyRollingFile
    file: ${path.logs}/${cluster.name}-access.log
    datePattern: "'.'yyyy-MM-dd"
    layout:
      type: pattern
      conversionPattern: "[%d{ISO8601}] %m%n"

As can be seen above, by default audit information is appended to the access.log file located in the standard elasticsearch logs directory (typically located at $ES_HOME/logs).

« IP filtering Integrating Shield with clients »