- Observability: other versions:
- What is Elastic Observability?
- What’s new in 8.7
- Send data to Elasticsearch
- Spin up the Elastic Stack
- Deploy Elastic Agent to send data
- Deploy Beats to send data
- Elastic Serverless Forwarder for AWS
- Deploy serverless forwarder
- Configuration options
- Troubleshooting
- Observability overview page
- Application performance monitoring (APM)
- Application logs
- Log monitoring
- Infrastructure monitoring
- Uptime
- Synthetics (beta)
- Get started
- Scripting browser monitors
- Configure lightweight monitors
- Manage monitors
- Analyze monitor data
- Monitor resources on private networks
- Use the CLI
- Configure projects
- Configure Synthetics settings
- Grant users access to secured resources
- Manage data retention
- Use Synthetics with traffic filters
- Migrate from the Elastic Synthetics integration
- User Experience
- Universal Profiling
- Alerting
- Cases
- CI/CD observability
- Troubleshooting
- Fields reference
- Tutorials
- Monitor Amazon Web Services (AWS) with Elastic Agent
- Monitor Amazon Web Services (AWS) with Beats
- Monitor Google Cloud Platform
- Monitor a Java application
- Monitor Kubernetes
- Monitor Microsoft Azure with Elastic Agent
- Monitor Microsoft Azure with the native Azure integration
- Monitor Microsoft Azure with Beats
APM agent ECS reformatting
editAPM agent ECS reformatting
editElastic APM agents can automatically reformat application logs to Elastic Common Schema (ECS) format without needing to add an ECS logger dependency or modify the application.
Requirements
- The Elastic APM agent for your programming language
- Filebeat configured to monitor and capture application logs
Pros
All the benefits of using ECS logging, without having to modify the application or its configuration:
- Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
- Decently human-readable JSON structure
- APM log correlation
Cons
- Requires an Elastic APM agent
- Not all APM agents support this feature
Supported APM agents/languages
- Ruby
- Python
- Java
Step 1: Enable APM agent reformatting
editEnable APM agent ECS reformatting in the supported APM agent:
See the Ruby or Java agent docs.
Step 2: Set up Filebeat
edit- Follow the Filebeat quick start to learn how to install Filebeat and connect to the Elastic Stack.
-
Add the following configuration to your
filebeat.yamlfile to start collecting log data.filebeat.yaml.
filebeat.inputs: - type: filestream paths: /path/to/logs.json parsers: - ndjson: overwrite_keys: true add_error_key: true expand_keys: true fields: service.name: your_service_name service.version: your_service_version service.environment: your_service_environment processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~
Use the filestream input to read lines from active log files.
Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.
Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.
The
service.name(required),service.version(optional) andservice.environment(optional) of the service you’re collecting logs from, used for Log correlation.Processors enhance your data. See processors to learn more.
- Make sure your application logs to stdout/stderr.
- Follow the Run Filebeat on Kubernetes guide.
-
Enable hints-based autodiscover (uncomment the corresponding section in
filebeat-kubernetes.yaml). -
Add these annotations to your pods that log using ECS-compatible JSON. This will make sure the logs are parsed appropriately.
annotations: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true
Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.
Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.
- Make sure your application logs to stdout/stderr.
- Follow the Run Filebeat on Docker guide.
- Enable hints-based autodiscover.
-
Add these labels to your containers that log using ECS-compatible JSON. This will make sure the logs are parsed appropriately.
docker-compose.yml.
labels: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true
Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.
Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.
Step 3: View your logs in Kibana
editUse the APM or Logs UI to search, filter, and visualize your logs.
On this page