Application logs

edit

Application logs provide valuable insight into events that have occurred within your services and applications.

Plaintext logs vs. ECS logs

edit

Logs are typically produced as either plaintext or structured. Plaintext logs contain only text and have no special formatting, for example:

2019-08-06T12:09:12.375Z INFO:spring-petclinic: Tomcat started on port(s): 8080 (http) with context path, org.springframework.boot.web.embedded.tomcat.TomcatWebServer
2019-08-06T12:09:12.379Z INFO:spring-petclinic: Started PetClinicApplication in 7.095 seconds (JVM running for 9.082), org.springframework.samples.petclinic.PetClinicApplication
2019-08-06T14:08:40.199Z DEBUG:spring-petclinic: init find form, org.springframework.samples.petclinic.owner.OwnerController

Structured logs, on the other hand, follow a predefined, repeatable pattern or structure. This structure is applied at write time—​preventing the need for parsing at ingest time. The Elastic Common Schema (ECS) defines a common set of fields to use when structuring logs. This structure allows logs to be easily ingested, and provides the ability to correlate, search, and aggregate on individual fields within your logs.

For example, the same example shown above might look like this when structured with ECS-compatable JSON:

{"@timestamp":"2019-08-06T12:09:12.375Z", "log.level": "INFO", "message":"Tomcat started on port(s): 8080 (http) with context path ''", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer"}
{"@timestamp":"2019-08-06T12:09:12.379Z", "log.level": "INFO", "message":"Started PetClinicApplication in 7.095 seconds (JVM running for 9.082)", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.samples.petclinic.PetClinicApplication"}
{"@timestamp":"2019-08-06T14:08:40.199Z", "log.level":"DEBUG", "message":"init find form", "service.name":"spring-petclinic","process.thread.name":"http-nio-8080-exec-8","log.logger":"org.springframework.samples.petclinic.owner.OwnerController","transaction.id":"28b7fb8d5aba51f1","trace.id":"2869b25b5469590610fea49ac04af7da"}

Log correlation

edit

To correlate your logs, you must annotate your logs with the APM identifier, service.name. Elastic APM integrates with the most popular logging frameworks in each programming language to accomplish this automatically.

Correlating your application logs with trace events allows you to:

  • view the context of a log and the parameters a user provided
  • view all logs belonging to a particular trace
  • easily move between logs and traces when debugging application issues in Kibana

Learn more about log correlation in the APM Guide: log correlation, or in any of the agent-specific ingestion guides:

How to ingest application logs

edit

There are four main ways to ingest application logs into the Elastic Stack. Your use case will help determine the method that’s right for you.

Plaintext logs with Filebeat

edit

Use Filebeat to parse and ingest raw, plain-text application logs.

Requirements

  • (Optional) Elastic APM agent for your programming language (for log correlation)
  • Raw, plain-text application logs stored on the file system
  • Filebeat configured to monitor and capture application logs

Pros

  • All programming languages/frameworks are supported
  • Existing application logs can be ingested
  • Does not require modification of the application or its configuration, unless log correlation is required

Cons

  • Must parse application logs to be useful—meaning writing and maintaining Grok patterns and spending CPU cycles on parsing
  • Parsing is tied to the application log format, meaning it can differ per application and needs to be maintained over time
  • Log correlation requires modifying the application log format and inject IDs in log messages

Learn more: Plaintext logs with Filebeat.

ECS loggers with Filebeat

edit

Elastic Common Schema (ECS) loggers format your logs into ECS-compatible JSON, removing the need to manually parse logs.

Requirements

  • (Optional) Elastic APM agent for your programming language (for log correlation)
  • The Elastic ECS logger for your language or framework
  • Filebeat configured to monitor and capture application logs

Pros

  • Popular logging frameworks supported
  • Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
  • Decently human-readable JSON structure
  • APM log correlation
  • Resilient in case of outages

Cons

  • Not all frameworks are supported
  • Requires modification of the application and its log configuration

Learn more: ECS logging with Filebeat

ECS reformatting with Filebeat

edit

Elastic APM agents can automatically reformat application logs to Elastic Common Schema (ECS) format without needing to add an ECS logger dependency or modify the application.

Requirements

  • The Elastic APM agent for your programming language
  • Filebeat configured to monitor and capture application logs

Pros

All the benefits of using ECS logging, without having to modify the application or its configuration:

  • Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
  • Decently human-readable JSON structure
  • APM log correlation

Cons

  • Requires an Elastic APM agent
  • Not all APM agents support this feature

Supported APM agents/languages

Learn more: APM agent ECS reformatting with Filebeat

APM agent log sending

edit

Elastic APM agents can automatically capture and send logs directly to APM Server—​enabling you to easily ingest log events without needing a separate log shipper like Filebeat.

Requirements

The Elastic APM agent for your programming language.

Pros

  • Simple to set up as it only relies on the APM agent
  • No modification of the application required
  • No need to deploy Filebeat
  • No need to store log files in the file system.

Cons

  • Experimental feature
  • Limited APM agent support
  • Not resilient to outages: Log messages can be dropped when buffered in the agent or in APM Server

Supported APM agents/languages

Learn more: APM agent log sending