Using Marvel with Shield

Marvel consists of a user interface over a data exporter known as the agent. The agent runs on each node and accesses that node’s monitoring API. The agent can store this collected data locally, on the cluster, or send the data to an external monitoring cluster. Users can view and analyze the collected data with the Marvel UI.

To work with the Shield plugin, Marvel’s configuration needs to be adapted for the production cluster, which is the cluster being monitored, as well as the monitoring cluster, where the monitoring data is stored. For clusters that store their own monitoring data, apply both sets of settings to the single, production cluster.

You will configure at least two users to work with Marvel. These users have to exist on the monitoring cluster.

The default roles.yml file includes definitions for these two roles. The steps below show you how to create these users on the monitoring cluster.

The monitoring cluster is used to both store and view the Marvel data. When configuring Shield, you need to perform the following actions:

This is in the default roles.yml

marvel_agent:
  cluster: indices:admin/template/get, indices:admin/template/put
  indices:
    '.marvel-*': indices:data/write/bulk, create_index

marvel_user:
  cluster: cluster:monitor/nodes/info, cluster:admin/plugin/license/get
  indices:
    '.marvel-*': all

Once the roles are configured, create a user for the agent:

bin/shield/esusers useradd marvel_export -p strongpassword -r marvel_agent

Then create one or more users for the Marvel UI:

bin/shield/esusers useradd USER -p strongerpassword -r marvel_user

The Marvel agent is installed on every node in the production cluster. The agent collects monitoring data from the production cluster and stores the data on the monitoring cluster. The agent’s configuration specifies a list of hostname and port combinations for access to the monitoring cluster.

When the monitoring cluster uses the Shield plugin and is configured to accept only HTTPS requests, you must configure the agent on the production cluster to use HTTPS instead of the default HTTP protocol.

Authentication and protocol configuration are both controlled by the marvel.agent.exporter.es.hosts setting in the node’s elasticsearch.yml file. The setting accepts a list of monitoring cluster servers to serve as a fallback in case a server is unavailable. Each of these servers must be properly configured, as in the following example:

Example marvel.agent.exporter.es.hosts setting

marvel.agent.exporter.es.hosts: [ "https://USER:PASSWORD@node01:9200", "https://USER:PASSWORD@node02:9200"]

When the monitoring cluster uses HTTPS, the Marvel agent will attempt to validate the certificate of the Elasticsearch node in the monitoring cluster. If you are using your own CA you should specify a trust store that has the signing certificate of the CA. Here is an example config for the marvel.agent.exporter.es.truststore.* settings:

marvel.agent.exporter.es.hosts: [ "https://USER:PASSWORD@node01:9200", "https://USER:PASSWORD@node02:9200"]
marvel.agent.exporter.es.ssl.truststore.path: FULL_FILE_PATH
marvel.agent.exporter.es.ssl.truststore.password: PASSWORD

See the Marvel documentation for more details about other SSL related settings.

The Marvel UI supports SSL without the need for any additional configuration. You can change URL access scheme for Marvel to HTTPS.

Users attempting to access the Marvel UI with the URL https://HOST:9200/_plugin/marvel must provide valid credentials. See Monitoring Cluster settings for information on the required user configuration.

Sense also supports HTTPS access. Users that access Sense over URLs of the form https://host:9200/_plugin/marvel/sense/index.html must provide valid credentials if they have not already authenticated to a dashboard.

Users connecting to the production cluster with Sense must provide valid credentials. Clusters must be configured to enable cross-origin requests to enable users to connect with Sense. See the CORS documentation for details.