IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Programming language clients Log retention »

›

Logs and logging

edit

Logs and logging

edit

Your Enterprise Search deployment produces a range of logs, including crawler, API, and audit logs. What gets logged, how it is logged, and where you can find these logs, differs between Elastic Cloud and self-managed deployment options.

This guide will give you a high-level overview of Enterprise Search logging:

Manage your logs: Learn how to manage your logs, to ensure you have the information you need when you need it.
Logs quick reference: Jump here for a handy overview of the various Enterprise Search log types, including the data views (or index patterns) you need to find them.

Read this guide to learn how to create data views, and use two Kibana tools, Logs and Discover, to view and query your logs.

Manage your logs

edit

Your deployment writes some logs directly to disk, and others to Elasticsearch. For the full list, see the Logs quick reference.

Managing logs on Elastic Cloud deployments

edit

Elastic Cloud users can set up logs and monitoring in their Cloud environment, and redirect logs and metrics to the same or a different cluster.

Follow these steps to enable logs and metrics:

Navigate to Deployments → Deployment → Logs and metrics → Enable.
Create a new monitoring deployment, or select an existing one.

Learn more about enabling logging and monitoring in Elastic Cloud.

Managing logs on self-managed deployments using Elasticsearch

edit

A subset of logs are indexed into Elasticsearch automatically, using Filebeat, as part of the Enterprise Search installation process. This does not include general logs such as app-server.log.

We encourage self-managed users to index these additional logs into Elasticsearch using a monitoring deployment. Apply the Elastic Enterprise Search Filebeat configuration to ingest Enterprise Search log files into Elasticsearch.

When logs are written to Elasticsearch you can manage log retention policies using index lifecycle management (ILM). Otherwise, you will have to manage the rollover and rotation of your logs yourself. The sooner you automate these policies, the better!

View self-managed deployment logs on-disk

edit

Self-managed deployments have access to additional logs that are otherwise not available to Elastic Cloud users. You’ll find these in the Logs quick reference below, where the Cloud Elasticsearch index pattern is marked N/A.

To view your log files stored on disk, the process varies by deployment type.

Tar, deb, and rpm package installs:

Find the logs in /log/dir.

Docker:

Use the command docker exec -it <container name> /bin/bash to launch a bash shell in the container. Find the logs in /log/dir.
Alternatively, the command docker logs <container-name> will show the stdout of Enterprise Search.

Elastic Cloud Kubernetes

Use the command kubectl exec --stdin --tty <pod-name> -- /bin/bash to launch a bash shell to a running Enterprise Search pod. Find the logs in the /var/log/enterprise-search directory in an Enterprise Search pod.
Alternatively, the command kubectl logs <pod-name> will show the stdout of an Enterprise Search pod.

Logs quick reference

edit

The following table contains key information about Enterprise Search logs, including log types, index patterns, and filenames:

The event.dataset value makes it easier to filter by events when querying your logs.
Filter event.dataset further by finding the Elasticsearch index patterns for your logs of interest in the table below. Use these index patterns when creating data views in Kibana.
This table specifies which Elastic Cloud index patterns are intended to be used on your Enterprise Search deployment, versus your monitoring deployment, and whether a given index pattern matches hidden data streams and indices.
Read this guide to learn how to create data views, and use the Kibana tools, Logs and Discover, to view and query your logs.

Log type	Elasticsearch index pattern (Self-managed)	Elasticsearch index pattern (Cloud)	Local filename	event.dataset	Description
API logs	`logs-enterprise_search.api-default`	`logs-enterprise_search.api-default` (Enterprise Search deployment)	`filebeat.log`	`api`	Enterprise Search API logs, including queries and inserts.
App Search analytics	`logs-app_search.analytics-default`	`logs-app_search.analytics-default` (Enterprise Search deployment)	`filebeat.log`	`app-search-analytics`	App Search Analytics logs, including click-through and query metrics.
Workplace Search analytics	`logs-workplace_search.analytics-default`	`logs-workplace_search.analytics-default` (Enterprise Search deployment)	`filebeat.log`	`workplace-search-analytics`	Workplace Search Analytics logs, including click-through and query metrics.
Application logs	N/A	`.ds-elastic-cloud-logs*` (monitoring deployment, hidden index)	`app-server.log`	`enterprise_search.server`	Main application logs, including API calls made to the server, errors and corresponding stack traces. When troubleshooting an issue, you will typically start here. Elasticsearch debug logs also appear here.
Audit logs	`logs-enterprise_search.audit-default`	`logs-enterprise_search.audit-default` (Enterprise Search deployment)	`audit.log`	`enterprise-search-audit`	Log of events across Enterprise Search, useful for security-related auditing.
Content sources events	`logs-workplace_search.content_events-default`	`logs-workplace_search.content_events-default` (Enterprise Search deployment)	`filebeat.log`	`workplace_search.content_events`	Workplace Search content sources events.
Web crawler logs	`logs-crawler-default`	`logs-crawler-default` (Enterprise Search deployment) `ds-elastic-cloud-logs*` (monitoring deployment, hidden index)	`crawler.log`	`enterprise_search.crawler`	Detailed App Search web crawler logs.
Filebeat process	N/A	N/A	`filebeat`	N/A	Use these logs to confirm that Filebeat is correctly pushing events to Elasticsearch.
JVM GC statistics	N/A	N/A	`gc.log`	N/A	Java Virtual Machine (JVM) garbage collection statistics.
Internal statistics	N/A	N/A	`stats.log`	N/A	Timing statistics for various internal processes.
System requests	N/A	`.ds-elastic-cloud-logs*` (monitoring deployment, hidden index)	`system.log`	`enterprise_search.system`	Contains web requests, including method, status code, and parameters used.
Internal worker jobs	N/A	`.ds-elastic-cloud-logs*` (monitoring deployment, hidden index)	`worker.log`	`enterprise_search.worker`	High-level worker job queue logs.
Search relevance suggestions events	`logs-app_search.search_relevance_suggestions-default`	`logs-app_search.search_relevance_suggestions-default` (Enterprise Search deployment)	`filebeat.log`	`search-relevance-suggestions, search-relevance-suggestions`	Created when Curations powered by Adaptive Relevance is enabled. Logs all adaptive relevance curation suggestion events.
Search relevance suggestions system logs	N/A	N/A	`search-relevance-suggestions.log`	N/A	Created when Curations powered by Adaptive Relevance is enabled. Logs all adaptive relevance curation suggestion details.

Apply the Elastic Enterprise Search Filebeat configuration to ingest Enterprise Search log files into Elasticsearch.

Learn more about logging in Enterprise Search

edit

For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Learn more in this guide.

App Search exposes several event logging settings through its log settings API and a corresponding section of the Settings dashboard. Read the Log settings guide for more details.

Learn about log retention and how to disable logging in App Search.

Read this guide to learn how to create data views, and use two Kibana tools, Logs and Discover, to view and query your logs.

« Programming language clients Log retention »