Appendix 7. Trouble Shooting

edit

settings

edit
Some settings are not returned via the nodes settings API

This is intentional. Some of the settings are considered to be highly sensitive (e.g. all ssl settings, ldap bind_dn, bind_password and hostname_verification). For this reason, we filter these settings and not exposing them via the nodes info API rest endpoint. It is also possible to define additional sensitive settings that should be hidden using the shield.hide_settings setting:

shield.hide_settings: shield.authc.realms.ldap1.url, shield.authc.realms.ad1.*

The snippet above will also hide the url settings of the ldap1 realm and all settings of the ad1 realm.

esusers

edit
I configured the appropriate roles and the users, but I still get an authorization exception

Verify that the role names associated with the users match the roles defined in the roles.yml file. You can use the esusers tool to list all the users. Any unknown roles are marked with *.

esusers list
rdeniro        : admin
alpacino       : power_user
jacknich       : marvel,unknown_role* 

unknown_role was not found in roles.yml

Active Directory

edit
Certain users are being frequently locked out of Active Directory

Check your realm configuration; realms are checked serially, one after another. If your Active Directory realm is being checked before other realms and there are usernames that appear in both Active Directory and another realm, a valid login for one realm may be causing failed login attempts in another realm.

For example, if UserA exists in both Active Directory and esusers, and the Active Directory realm is checked first and esusers is checked second, an attempt to authenticate as UserA in the esusers realm would first attempt to authenticate against Active Directory and fail, before successfully authenticating against the esusers realm. Because authentication is verified on each request, the Active Directory realm would be checked - and fail - on each request for UserA in the esusers realm. In this case, while the Shield request completed successfully, the account on Active Directory would have received several failed login attempts, and that account may become temporarily locked out. Plan the order of your realms accordingly.

Also note that it is not typically necessary to define multiple Active Directory realms to handle domain controller failures. When using Microsoft DNS, the DNS entry for the domain should always point to an available domain controller.

LDAP

edit
I can authenticate to LDAP, but I still get an authorization exception

A number of configuration options can cause this error.

group identification

Groups are located by either an LDAP search or by the "memberOf" attribute on the user. Also, If subtree search is turned off, it will search only one level deep. See the LDAP Settings for all the options. There are many options here and sticking to the defaults will not work for all scenarios.

group to role mapping

Either the role_mapping.yml file or the location for this file could be misconfigured. See Shield Files for more.

role definition

Either the roles.yml file or the location for this file could be misconfigured. See Shield Files for more.

To help track down these possibilities, add shield.authc: DEBUG to the logging.yml config file. A successful authentication should produce debug statements that list groups and role mappings.

Encryption & Certificates

edit
curl on the Mac returns a certificate verification error even when the --cacert option is used

Apple’s integration of curl with their keychain technology disables the --cacert option. See http://curl.haxx.se/mail/archive-2013-10/0036.html for more information.

You can use another tool, such as wget, to test certificates. Alternately, you can add the certificate for the signing certificate authority MacOS system keychain, using a procedure similar to the one detailed at the Apple knowledge base. Be sure to add the signing CA’s certificate and not the server’s certificate.

SSLHandshakeException causing connections to fail

edit

A SSLHandshakeException will cause a connection to a node to fail and indicates that there is a configuration issue. Some of the common exceptions are shown below with tips on how to resolve these issues.

java.security.cert.CertificateException: No name matching node01.example.com found

Indicates that a client connection was made to node01.example.com but the certificate returned did not contain the name node01.example.com. In most cases, the issue can be resolved by ensuring the name is specified as a SubjectAlternativeName during certificate creation. Another scenario is when the environment does not wish to use DNS names in certificates at all. In this scenario, all settings in elasticsearch.yml should only use IP addresses and the following setting needs to be set in elasticsearch.yml:

shield.ssl.hostname_verification.resolve_name: false
java.security.cert.CertificateException: No subject alternative names present

Indicates that a client connection was made to an IP address but the returned certificate did not contain any SubjectAlternativeName entries. IP addresses are only used for hostname verification if they are specified as a SubjectAlternativeName during certificate creation. If the intent was to use IP addresses for hostname verification, then the certificate will need to be regenerated. Also verify that shield.ssl.hostname_verification.resolve_name: false is not set in elasticsearch.yml.

javax.net.ssl.SSLHandshakeException: null cert chain and javax.net.ssl.SSLException: Received fatal alert: bad_certificate

The SSLHandshakeException above indicates that a self-signed certificate was returned by the client that is not trusted as it cannot be found in the truststore or keystore. The SSLException above is seen on the client side of the connection.

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target and javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

The SunCertPathBuilderException above indicates that a certificate was returned during the handshake that is not trusted. This message is seen on the client side of the connection. The SSLException above is seen on the server side of the connection. The CA certificate that signed the returned certificate was not found in the keystore or truststore and needs to be added to trust this certificate.

Other SSL/TLS related exceptions

edit

The are other exceptions related to SSL that may be seen in the logs. Below you will find some common exceptions and their meaning.

java.lang.IllegalArgumentException: Cannot support TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA with currently installed providers

This error can occur when using the Shield defaults as some distributions of OpenJDK do not enable the PKCS11 provider. With Shield 1.0, this can prevent a node from starting. Using Shield 1.1 or later will allow the node to start up. If it is not possible to use Shield 1.1, the following setting in elasticsearch.yml will workaround the issue:

shield.ssl.ciphers: ["TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_DHE_RSA_WITH_AES_128_CBC_SHA"]

For the best performance and security, enabling the PKCS11 provider is recommended. Please consult your JVM documentation for details on enabling the PKCS11 provider. Once the PKCS11 provider is enabled, shield.ssl.ciphers no longer needs to be specified in elasticsearch.yml.

org.elasticsearch.common.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

Indicates that there was incoming plaintext traffic on an SSL connection. This typically occurs when a node is not configured to use encrypted communication and tries to connect to nodes that are using encrypted communication or when a plain http request is made to a node that is expecting https traffic. Please verify that all nodes are using the same setting for shield.transport.ssl and that external applications are using https when using the REST interface.

java.io.StreamCorruptedException: invalid internal transport message format, got

Indicates an issue with data received on the transport interface in an unknown format. This can happen when a node with encrypted communication enabled connects to a node that has encrypted communication disabled. Please verify that all nodes are using the same setting for shield.transport.ssl.

java.lang.IllegalArgumentException: empty text

The exception is typically seen when a https request is made to a node that is not using https. If https is desired, please ensure the following setting is in elasticsearch.yml:

shield.http.ssl: true

Exceptions when unlicensed

edit
WARN: Failed to execute IndicesStatsAction for ClusterInfoUpdateJob

This warning occurs in the logs every 30 seconds when the Shield license is expired or invalid. It is caused by a periodic internal request to gather disk usage information from the nodes and indices, to enable disk-based shard allocation. Disk-based shard allocation is not required, though it is enabled by default.

If you are using elasticsearch 1.4.3 or higher with disk-based shard allocation enabled, it will be automatically disabled when the Shield license is expired or invalid, and will be automatically re-enabled when a valid Shield license is installed.

If you are using elasticsearch 1.4.2 with disk-based shard allocation enabled, we recommend manually disabling disk-based shard allocation while your Shield license is expired, and re-enabling it after installing a valid Shield license. Instructions for disabling disk-based shard allocation are here.