Get started
editGet started
editStep 1: Set up application logging
editAdd the dependency
editAdd this line to your application’s Gemfile:
gem 'ecs-logging'
Execute with:
bundle install
Alternatively, you can install the package yourself with:
gem install ecs-logging
Configure
editEcs::Logger
is a subclass of Ruby’s own Logger
and responds to the same methods.
For example:
require 'ecs_logging/logger' logger = EcsLogging::Logger.new($stdout) logger.info('my informative message') logger.warn { 'be aware that…' } logger.error('a_progname') { 'oh no!' }
Logs the following JSON to $stdout
:
{"@timestamp":"2020-11-24T13:32:21.329Z","log.level":"INFO","message":"very informative","ecs.version":"1.4.0"} {"@timestamp":"2020-11-24T13:32:21.330Z","log.level":"WARN","message":"be aware that…","ecs.version":"1.4.0"} {"@timestamp":"2020-11-24T13:32:21.331Z","log.level":"ERROR","message":"oh no!","ecs.version":"1.4.0","process.title":"a_progname"}
Additionally, it allows for adding additional keys to messages.
For example:
logger.info('ok', labels: { my_label: 'value' }, 'trace.id': 'abc-xyz')
Logs the following:
{ "@timestamp":"2020-11-24T13:32:21.331Z", "log.level":"INFO", "message":"oh no!", "ecs.version":"1.4.0", "labels":{"my_label":"value"}, "trace.id":"abc-xyz" }
To include info about where the log was called, call the methods with include_origin: true
,
like logger.warn('Hello!', include_origin: true)
. This logs:
{ "@timestamp":"2020-11-24T13:32:21.331Z", "log.level":"WARN", "message":"Hello!", "ecs.version":"1.4.0", "log.origin": { "file.line": 123, "file.name": "my_file.rb", "function": "call" } }
Rack configuration
edituse EcsLogging::Middleware, $stdout
Example output:
{ "@timestamp":"2020-12-07T13:44:04.568Z", "log.level":"INFO", "message":"GET /", "ecs.version":"1.4.0", "client":{ "address":"127.0.0.1" }, "http":{ "request":{ "method":"GET", "body.bytes":"0" } }, "url":{ "domain":"example.org", "path":"/", "port":"80", "scheme":"http" } }
Step 2: Enable APM log correlation (optional)
editIf you are using the Elastic APM Ruby agent, enable log correlation.
Step 3: Configure Filebeat
edit- Follow the Filebeat quick start
-
Add the following configuration to your
filebeat.yaml
file.
For Filebeat 7.16+
filebeat.yaml.
filebeat.inputs: - type: filestream paths: /path/to/logs.json parsers: - ndjson: overwrite_keys: true add_error_key: true expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~
Use the filestream input to read lines from active log files. |
|
Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts. |
|
Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors. |
|
Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure. |
|
Processors enhance your data. See processors to learn more. |
For Filebeat < 7.16
filebeat.yaml.
filebeat.inputs: - type: log paths: /path/to/logs.json json.keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~
- Make sure your application logs to stdout/stderr.
- Follow the Run Filebeat on Kubernetes guide.
-
Enable hints-based autodiscover (uncomment the corresponding section in
filebeat-kubernetes.yaml
). - Add these annotations to your pods that log using ECS loggers. This will make sure the logs are parsed appropriately.
annotations: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true
Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts. |
|
Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors. |
|
Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure. |
- Make sure your application logs to stdout/stderr.
- Follow the Run Filebeat on Docker guide.
- Enable hints-based autodiscover.
- Add these labels to your containers that log using ECS loggers. This will make sure the logs are parsed appropriately.
docker-compose.yml.
labels: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true
Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts. |
|
Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors. |
|
Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure. |
For more information, see the Filebeat reference.