Get started

edit

Step 1: Set up application logging

edit

Add the dependency

edit

Add this line to your application’s Gemfile:

gem 'ecs-logging'

Execute with:

bundle install

Alternatively, you can install the package yourself with:

gem install ecs-logging

Configure

edit

Ecs::Logger is a subclass of Ruby’s own Logger and responds to the same methods.

For example:

require 'ecs_logging/logger'

logger = EcsLogging::Logger.new($stdout)
logger.info('my informative message')
logger.warn { 'be aware that…' }
logger.error('a_progname') { 'oh no!' }

Logs the following JSON to $stdout:

{"@timestamp":"2020-11-24T13:32:21.329Z","log.level":"INFO","message":"very informative","ecs.version":"1.4.0"}
 {"@timestamp":"2020-11-24T13:32:21.330Z","log.level":"WARN","message":"be aware that…","ecs.version":"1.4.0"}
 {"@timestamp":"2020-11-24T13:32:21.331Z","log.level":"ERROR","message":"oh no!","ecs.version":"1.4.0","process.title":"a_progname"}

Additionally, it allows for adding additional keys to messages.

For example:

logger.info('ok', labels: { my_label: 'value' }, 'trace.id': 'abc-xyz')

Logs the following:

{
  "@timestamp":"2020-11-24T13:32:21.331Z",
  "log.level":"INFO",
  "message":"oh no!",
  "ecs.version":"1.4.0",
  "labels":{"my_label":"value"},
  "trace.id":"abc-xyz"
}

To include info about where the log was called, call the methods with include_origin: true, like logger.warn('Hello!', include_origin: true). This logs:

{
  "@timestamp":"2020-11-24T13:32:21.331Z",
  "log.level":"WARN",
  "message":"Hello!",
  "ecs.version":"1.4.0",
  "log.origin": {
    "file.line": 123,
    "file.name": "my_file.rb",
    "function": "call"
  }
}

Rack configuration

edit
use EcsLogging::Middleware, $stdout

Example output:

{
  "@timestamp":"2020-12-07T13:44:04.568Z",
  "log.level":"INFO",
  "message":"GET /",
  "ecs.version":"1.4.0",
  "client":{
    "address":"127.0.0.1"
  },
  "http":{
    "request":{
      "method":"GET",
      "body.bytes":"0"
    }
  },
  "url":{
    "domain":"example.org",
    "path":"/",
    "port":"80",
    "scheme":"http"
  }
}

Step 2: Enable APM log correlation (optional)

edit

If you are using the Elastic APM Ruby agent, enable log correlation.

Step 3: Configure Filebeat

edit
  1. Follow the Filebeat quick start
  2. Add the following configuration to your filebeat.yaml file.

For Filebeat 7.16+

filebeat.yaml.

filebeat.inputs:
- type: filestream 
  paths: /path/to/logs.json
  parsers:
    - ndjson:
      overwrite_keys: true 
      add_error_key: true 
      expand_keys: true 

processors: 
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Use the filestream input to read lines from active log files.

Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.

Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.

Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.

Processors enhance your data. See processors to learn more.

For Filebeat < 7.16

filebeat.yaml.

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~

For more information, see the Filebeat reference.