Enable logging and monitoring

The deployment logging and monitoring feature lets you monitor your deployment in Kibana by shipping logs and metrics to a monitoring deployment. You can:

Monitoring consists of two components:

The steps in this section cover only the enablement of the monitoring and logging features in Elastic Cloud Enterprise. For more information on how to use the monitoring features, refer to the following documentation:

Some limitations apply when you use monitoring on Elastic Cloud Enterprise. To learn more, check the monitoring restrictions and limitations.

For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into Elasticsearch and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable.

How many monitoring deployments you use depends on your requirements:

Logs and metrics that get sent to a dedicated monitoring Elasticsearch deployment may not be cleaned up automatically and might require some additional steps to remove excess data periodically.

When you enable monitoring in Elastic Cloud Enterprise, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. The retention period is configured in the .monitoring-8-ilm-policy index lifecycle policy. To view or edit the policy open Kibana Stack management > Data > Index Lifecycle Policies.

When you enable self-monitoring in Elastic Cloud Enterprise, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. Monitoring data is retained for three days by default or as specified by the xpack.monitoring.history.duration user setting.

To retain monitoring indices as is without deleting them automatically, you must disable the cleaner service by adding a disabled local exporter in your cluster settings.

For example

PUT /_cluster/settings
{
    "persistent": {
        "xpack.monitoring.exporters.__no-default-local__": {
            "type": "local",
            "enabled": false
        }
    }
}

When you enable monitoring in Elastic Cloud Enterprise by configuring your deployment to send monitoring data to itself, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. The retention period when a deployment sends monitoring data to itself depends on the version of Elasticsearch:

When monitoring for production use, where you configure your deployments to send monitoring data to a dedicated monitoring deployment for indexing, this retention period does not apply. Monitoring indices on a dedicated monitoring deployment are retained until you remove them. There are three options open to you:

Log retention is determined by the version of the monitoring deployment.

When monitoring to a deployment at version 6.7 or later, you can configure Index Lifecycle Management (ILM) to manage retention of your monitoring and logging indices. When sending logs to a deployment at version 7.0 or later, an ILM policy is pre-configured to manage log retention and the policy can be customized to your needs.

When monitoring to a deployment before version 6.6, you can run Curator on-premise to manage indices on Elastic Cloud Enterprise. For example, you can use Curator to clean up monitoring indices like any other time-based index.

Elastic Cloud Enterprise manages the installation and configuration of the monitoring agent for you. When you enable monitoring on a deployment, you are configuring where the monitoring agent for your current deployment should send its logs and metrics.

To enable monitoring on your deployment:

With monitoring enabled for your deployment, you can access the logs and stack monitoring through Kibana.

Alternatively, you can access logs and metrics directly on the Kibana Logs and Stack Monitoring pages in the target monitoring deployment. You can also create an elastic-cloud-logs-* data view (formerly index pattern) to view your deployment’s logs in the Kibana Discover tab. Several fields are available for you to view logs based on key details, such as the source deployment:

When shipping logs to a monitoring deployment there are more logging features available to you. These features include:

After you’ve enabled log delivery on your deployment, you can add the Elasticsearch user settings to enable these features.

After you’ve enabled log delivery on your deployment, you can add the Kibana user settings to enable this feature.

Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments:

Enterprise Search

App Search

APM

Fleet and Elastic Agent

The ˆ*ˆ indicates that we also index the archived files of each type of log.

Check the respective product documentation for more information about the logging capabilities of each product.

With logging and monitoring enabled for a deployment, metrics are collected for Elasticsearch, Kibana, Enterprise Search, and APM with Fleet Server.

Audit logs are useful for tracking security events on your Elasticsearch and/or Kibana clusters. To enable Elasticsearch audit logs on your deployment:

A plan change will run on your deployment. When it finishes, audit logs will be delivered to your monitoring deployment.