Brewing in Beats: Read Redis slow logs with Filebeat

By
Monica Sarbu

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Filebeat: Read slow logs from Redis

The Redis Slow Log is a system to log queries that exceeded a configured execution time. The slow logs are accumulated in memory, so no files are written on the disk. To read the slow logs, you can use the SLOWLOG GET N command, and it will return only the N most recent slow log entries.

With this change, a new prospector type is added to Filebeat that connects to Redis, retrieves the slow logs and clears the buffer so on the next poll only new events will be read. It’s inspired from the community Beat, Rsbeat. This is the first specialized prospector type in Filebeat, which opens the door for other interesting use cases in the future.

With this configuration: 

- input_type: redis
  hosts: ["localhost:6379"]
   # How often to fetch the Redis slow logs
   #scan_frequency: 10s

   # Network type to be used for redis connection. Default: tcp
   #network: tcp
   …

Filebeat generates an event for each Redis slow log: 

{
  "@timestamp": "2017-05-16T06:27:17.000Z",
  "beat": {
    "hostname": "ruflin",
    "name": "ruflin",
    "read_timestamp": "2017-05-16T06:27:19.275Z",
    "version": "6.0.0-alpha2"
  },
  "message": "SET hello world",
  "redis": {
    "slowlog": {
      "args": [
        "world"
      ],
      "cmd": "SET",
      "duration": {
        "us": 11
      },
      "id": 38,
      "key": "hello"
    }
  }
}

Where the parsed slow log is available under redis and the message contains the full Redis command, including the arguments concatenated.

This feature is experimental, and will be included in the 6.0.0 version.

Repository: elastic/beats

Filebeat

Changes in master:

  • Fix panic on empty multiline pattern #4377
  • Add log message when multline event flushed #4375
  • Make state comparison more generic #4190
  • Filebeat Redis prospector type #4180
Metricbeat

Changes in master:

  • Remove unnecessary print statement in schema apis #4355
  • Change Metricbeat default config to reduce disk space #4329
Packetbeat

Changes in master:

  • Do not activate memcache by default (#4335) #4389
  • Fix parsing of interface options with _ (#4334) #4378
Infrastructure

Changes in master:

  • Upgrade Go to 1.8.3 #4401
  • Add a CSV summary for the dependencies #4395
  • Add versions to the NOTICE file #4383
  • Improve `make notice` to work on libs not 3 dirs long #4380
  • Sync vendor directory #4372
  • Python 3 support #4346

Changes in 5.4:

  • Use .go-version to specify the Go version for all CI builds #4303
  • Upgrade to Go 1.7.6 #4400
Documentation

Changes in 5.4:

  • Cherrypick community beats into 5.4 #4387

Changes in 5.3:

  • Revert "Bump docs version to 5.3.3" #4399
  • Bump docs version to 5.3.3 #4392

Changes in master:

  • Add simple examples that illustrate multiline settings #4408
  • Fix typos in community beats #4388
  • Add docs for Docker images #4312
Packaging

Changes in master:

  • Add non-root support in sysvinit script #4340