Brewing in Beats: Filebeat module generator

By
Monica Sarbu

Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

What's new in Beats

Filebeat Modules code generator

The new generate sub-command has been added to filebeat in #9314. The generate sub command lets you generate the required files for a new filebeat module or a new fileset within a module. 

$ ./filebeat generate -h
Generate Filebeat modules, filesets and fields.yml
Usage:
  filebeat generate [command]
Available Commands:
  fields      Generates a new fields.yml file for fileset
  fileset     Generates a new fileset
  module      Generates a new module

The subcommands module, fileset and fields use the same code as the scripts.

Filebeat registry changes

The filebeat registry is now stored in ${path.data}/registry/filebeat/data.json by default (See: #10504). The new directory also contains a meta.json file with version number, so to better support migrations between registry file format changes in the future. The data file’s contents and encoding is not changed. Filebeat will migrate the old registry file to the new layout on start. The location of an old registry file in a non-standard location can be configured via filebeat.registry.migrate_file. The settings for the filebeat registry have been moved into its own namespace as well. The filebeat.registry_file setting has been replaced by filebeat.registry.path, now configuring the directory. The settings filebeat.registry_flushand filebeat.registry_file_permission have been renamed to filebeat.registry.flush and filebeat.registry.file_permission.

What's new in Central Configuration

The CM team has been hard at work to get the Kibana CM UI refactor in and backported to 6.x; this PR will allow CM to scale better with a large number of beats and was a critical piece to get in to make future improvements easier to implement. Since we are on the road to GA, we have started to create scripts to stress test the UI and the API in a context of tens of thousands of beats and also tens of thousands of configuration; this testing also exposed a few issues.

Screen Shot 2019-02-05 at 7.38.34 AM.png

What's new in Stack monitoring

In 7.0 Elasticsearch is deprecating most _xpack/* endpoints. This work was completed in beats/#9656. A critical bug was found and fixed which prevented nodes from appearing in the node listing page under certain conditions. A full breakdown of this issue is found in beats/#10639.

The monitoring UI now correctly detects and respects a cluster configuration regarding CCR and this issue is outlined in kibana/#28840. Finally, a number of smaller regressions and bug fixes were addressed. A complete list can be found here.

The Stack Monitoring team continues its push toward the ability to display stack logs inside the Stack Monitoring application and this week wrapped up the bulk of the back-end work necessary to achieve this. It's possible now to ship and parse Elasticsearch slow logs, deprecation logs, and audit logs using Filebeat. (beats/#10385, beats/#10414, beats/#10445, and beats/#10447.) .

This largely completes the needed back-end work for log displays and the focus now turns to the implementation of display elements which will begin to stream relevant logs into various areas of the Stack Monitoring web interface.

All changes

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Bump default beat version to 8.0.0 #10626
  • Add username password back from es to kibana config #10553
  • Disable migration aliases in index pattern #10478
  • Fix stopping of modules started by kubernetes autodiscover #10476
  • Add time encoders to disk spool #10197
  • Skipping more CM system tests for now #10481
  • Do not use include_type_name=false when talking to ES 6.x #10444
  • vendor: update rcrowley/go-metrics #10381
  • Update go-ucfg to 0.7.0 #10363
  • Refactor index management #10347
  • Mark indices with the beat name #10313
  • Allow to remove setup cmd flags from beats. #10277
  • Remove deprecated --setup, --version, and --configflags CLI flags #10138

Changes in 6.x:

  • Do not panic when no tokenizer string is configured for dissect #8928
  • Add logging trace at debug level for the pipeline client. #9016
  • Update go-txfile 0.0.6 #10289
    • Update output and template generation to type removal in ES API #10156
Metricbeat

Changes in master:

  • Remove filter in DescribeInstances #10628
  • [Metricbeat] Skip TestMysql80 and TestPercona80 tests #10609
  • Update and add missing descriptions to ETCD module #10592
  • [Metricbeat] Update RabbitMQ module to use reporter interface #10560
  • Metricbeat MSSQL module: Fix for Transaction log flaky test that was sometimes failing #10480
  • [Metricbeat] Release Golang module as GA. #10312
  • Replace deprecated _xpack endpoints #9656
  • Move convertPeriodToDuration and getRegions into aws.go #10474
  • Fix camel-cased fields in kubernetes pod metricset #10470
  • Support Kafka 2.1.0 #10440
  • Metricbeat overview dashboard for Zookeeper #10379
  • [Metricbeat] Add data.json to dommemstat metricset (#10217) #10371
  • [Metricbeat] Migrate php_fpm to ECS #10366
  • [Metricbeat] Convert HAProxy to reporter metricset #10365
  • [Metricbeat] Change type of field docker.container.ip_addresses to ip instead of keyword #10364
  • Update Nats module docs #10359
  • New functions to close a mysql connection. #10355
  • [Metricbeat] Release AWS as GA #10345
  • server Metricset for Zookeeper Metricbeat module #10341
  • Add state metrics for each EC2 instance #10334
  • Add mapping for munin and options to override service type and name #10322
  • [Metricbeat] Move redis.info metricset to ECS #10319
  • Change text fields to keyword for Metricbeat #10318
  • [Metricbeat] Release munin as GA. #10311
  • Add nats dashboard #10235
  • Add couchdb module #9406

Changes in 6.x:

  • Metricbeat overview dashboard for Zookeeper #10379
  • Add remaining memory metrics of pods in Kubernetes metricbeat module #10157
  • New functions to close a mysql connection. #10355
  • server Metricset for Zookeeper Metricbeat module #10341
    • Support MySQL 8, Percona and MariaDB in metricbeat #10261
Packetbeat

Changes in 6.6:

  • Update vendored tsg/gopacket #10477

Changes in master:

  • Update vendored tsg/gopacket #10477

Changes in 6.x:

  • Update vendored tsg/gopacket #10477
Filebeat

Changes in master:

  • [Filebeat] Skip flaky registrar tests #10607
  • [Filebeat] Fix postgres dashboard for event.duration #10604
  • Fix missing CHANGELOG for #10006 #10579
  • Make santa dashboard formatted JSON #10566
  • Missing CHANGELOG for #10006 #10565
  • Update filebeat registry configuration #10504
  • New Filebeat subcommand: generate #9314
  • Move dashboard to kibana/7 dir #10460
  • Remove dissect tokenizing from Traefik Filebeat Access Fileset #10442
  • Use lowercase pattern for years parsing in filebeat pipelines #10436
    • [Filebeat] Iptables / ubiquiti module #10176

Changes in 6.x:

  • [Filebeat] Remove Santa module directory accidentally backported to 6.x #10610
  • Fix missing CHANGELOG for #10006 #10579
  • [Filebeat] Iptables / ubiquiti module #10176
  • Use lowercase pattern for years parsing in filebeat pipelines #10436
Auditbeat

Changes in 6.7:

  • Add rpm packaging rebase #10429

Changes in 6.6:

  • [Auditbeat] Fix flaky TestRecursive test under Windows (#10424) #10425

Changes in master:

  • [Auditbeat] Skip flaky test_metricset_package test #10634
  • [Auditbeat] Socket: Remove ecsDirectionString() #10616
  • [Auditbeat] Changelog for #10511 #10576
  • Set `file.origin` type to keyword #10544
  • [Auditbeat] Login: Change event.type to event.kind #10512
  • [Auditbeat] System module dashboards #10511
  • [Auditbeat] System module: Detect package updates #10508
  • [Auditbeat] Read formula path from INSTALL_RECEIPT.json for Homebrew packages #10507
  • [Auditbeat] System module: Add entity_id fields #10500
  • [Auditbeat] Set up and remove data dir in all unit tests #10482
  • [Auditbeat] Auditd: Change user fields to ECS #10456
  • [Auditbeat] Auditd: Check all fields are in fields.yml in unit test #10457
  • Add rpm packaging rebase #10429
  • [Auditbeat] Fix flaky TestRecursive test under Windows (#10424) #10425
  • [Auditbeat] Make user.group.name optional in system test on Windows #10404
  • [Auditbeat] Enable System module config on Windows #10237
  • [Auditbeat] System module: Update and re-enable package dataset #10225
  • [Auditbeat] Add user information to processes #9963
  • [Auditbeat] Login metricset #9327

Changes in 6.x:

  • [Auditbeat] System module dashboards #10511
  • [Auditbeat] System module: Add entity_id fields #10500
  • [Auditbeat] System module: Detect package updates #10508
  • [Auditbeat] Read formula path from INSTALL_RECEIPT.json for Homebrew packages #10507
  • [Auditbeat] Login metricset #9327
  • [Auditbeat] Fix flaky TestRecursive test under Windows (#10424) #10425
  • [Auditbeat] System module: Update and re-enable package dataset #10225
  • [Auditbeat] Add user information to processes #9963
  • [Auditbeat] Enable System module config on Windows #10237
Heartbeat

Changes in master:

  • [Heartbeat] Populate url.port field for http(s) fields. #10467
  • [Heartbeat] Fix id/summary with multi-url configs #10408

Changes in 6.x:

  • [Heartbeat] Incorporate factory metadata for autodiscover #10258
Journalbeat

Changes in 6.6:

  • Migrate registry from previous incorrect path #10486

Changes in master:

  • Change type of text fields to keyword #10542
  • Do not read last entry when falling back to tail #10541
  • Fix typo in the field name `container.id_truncated` #10525
  • Fix Journalbeat dashboard on 7.x #10524
  • Migrate registry from previous incorrect path #10486
  • Do not stop collecting events when journal entries change #9994

Changes in 6.x:

  • Migrate registry from previous incorrect path #10486
  • Do not read last entry when falling back to tail #10541
  • Fix Journalbeat dashboard in 6.7 #10517
  • Do not stop collecting events when journal entries change #9994
Functionbeat

Changes in 6.6:

  • Add Journalbeat and Function in the readme.md #9238

Changes in master:

  • Remove experimental flags and mark most of the AWS provider trigger stable. #10564
  • Allow a keystore to be send with functionbeat files #10263

Changes in 6.x:

  • Remove experimental flags and mark most of the AWS provider trigger stable. #10564
  • Add Journalbeat and Function in the readme.md #9238
  • Allow a keystore to be send with functionbeat files #10263
  • Fix permissions issues for SQS #10265
  • [functionbeat] Allow Kinesis to deploy using the CLI. #10116
Dashboards

Changes in master:

  • Fix RabbitMQ dashboard #10603
  • Move iptables dashboards to kibana/7 #10496
  • Cleanup _exists_ queries in dashboards #10483
  • Metricbeat dashboard for MSSQL transaction log metricset #10310
  • Clean up Kibana, remove Elasticsearch loading and 5.x version #10451
Testing

Changes in master:

  • Revert Kibana snapshot fix #10521
  • Temporary fix to get CI back to green #10516
  • Updating stack versions to latest in docker images used for tests #10499
  • Allow testing of specific filesets #10443
  • Use default path for data.json if no other path is set #10402
  • Correctly bubble up errors when an integration test fails in a docker container. #10380
  • Support generation of multiple data files on metricbeat integration tests #10367
  • Fix port in couchdb module system test #10358

Changes in 6.x:

  • Disable CM for 6.x until snapshot is ready. #10533
  • Test 6.x against latest snapshot builds #10523
  • Updating stack versions to latest in docker images used for tests #10499
  • Allow testing of specific filesets #10443
  • Correctly bubble up errors when an integration test fails in a docker container. #10380
  • Increase script compilations rate for Filebeat system tests #10388

Changes in 6.6:

  • Correctly bubble up errors when an integration test fails in a docker container. #10380
  • Update testing env 6.6.0 #10390
Packaging

Changes in 6.6:

  • Update beats to fpm 1.11.0 #10527

Changes in master:

  • Update beats to fpm 1.11.0 #10527
  • Customize x-pack/metricbeat config for Windows #10439
  • Use packages from x-pack/metricbeat #10396

Changes in 6.x:

  • Update beats to fpm 1.11.0 #10527

Changes in 5.6:

Documentation

Changes in 6.7:

  • Update docs branches for 6.7 #10613

Changes in 6.6:

  • Add documentation about namespace option in kubernetes #10473
  • Bump docs version for 6.6.0 #10391
  • Close changelog for 6.6.0 #10389
  • Cherrypick community beats into 6.6 branch #10382

Changes in master:

  • Fix list formatting in redis key metricset #10580
  • [Docs] Explain how to set the data path for CM enrollment on windows #10503
  • Script to generate breaking field changes list #10405
  • Add documentation about namespace option in kubernetes #10473
  • [docs] Fix the doc build #10468
  • [docs]: Add beat_version_key attribute #10416
  • Add servicebeat to list of Community Beats #10406
  • Update configs for old beat.* fields #10370
  • [Docs]Fix Metricbeat docker example #10270
  • [Docs] APM additions and code cleanup #10187
  • Update newbeat.asciidoc to Reflect New generate.py #9052

Changes in 6.x:

  • Fix list formatting in redis key metricset #10580
  • Add documentation about namespace option in kubernetes #10473
  • Update autodiscover-hints.asciidoc #10084
  • Update newbeat.asciidoc to Reflect New generate.py #9052
  • Cherrypick community beats into 6.x branch #10383
ECS

Changes in 7.0:

  • Fix field alias in Winlogbeat template. #10622

Changes in master:

  • Fix field alias in Winlogbeat template. #10622
  • Fix JSON encoding issue with my Winlogbeat dashboard from #10333 #10586
  • Revert migration of haproxy frontend to process.name #10581
  • Rename auditd fields for ECS #10577
  • Suricata: Rename event.type to suricata.eve.event_type #10575
  • [Metricbeat] HAProxy info fields adjust to ECS #10568
  • [Metricbeat] Align rabbitmq with ECS and have module fields #10563
  • Rename `container.image.tag` to `container.log.tag` #10561
  • [Metricbeat] Update HAProxy module to follow ECS #10558
  • Change type of haproxy.source from text to keyword #10557
  • Change event.type to auditd.message_type #10536
  • Skip event.dataset and service.type if module is empty #10526
  • Address issue #10505 (add_kubernetes_metadata processor matcher not working) #10506
  • Finalize user_agent migration to ECS #10441
  • [Filebeat] kafka.log fileset change type of field class to keyword #10398
  • Migrate system socket fields metricset to ECS #10339
  • Migrate Winlogbeat to ECS, take 2 #10333
  • Introduce migration script for data in Kibana files #9998
  • Remove accidental overwrite of user_agent.device.name by user_agent.device #10472
  • Remove field url.hostname. #10469
  • Add `service.name` as an option to all Metricbeat modules #10427
  • Ensure source.address is always populated by the nginx module #10418
  • Change type of field backend_url and frontend_name in traefik.access #10401
  • [Filebeat] Change type from haproxy.log fileset fields from text to keyword #10397
  • [Filebeat] Replace Suricata/Eve fields with aliases to ECS fields #10377
  • [Winlogbeat] Fix duplicated type entry #10373
  • [Metricbeat] Move mongodb.status metricset to ECS #10368
  • Add event.kind and event.category #10357
  • Migrate system process metricset fields to ECS #10332
  • [Metricbeat] Rename http.request.body to http.request.body.content for ECS #10315
  • Convert the Filebeat auditd module to ECS #10192

Changes in 6.x:

  • Populate more ECS fields in the Suricata module #10006
  • Populate more ECS fields in the Suricata module #10006
Stack monitoring

Changes in master:

  • Ingest structured ES server logs #10428
  • Ingest structured ES slow logs #10447
  • Ingest structured ES deprecation logs #10445
  • [Filebeat] Changes to text fields in logstash module #10417
  • [Filebeat] Changes to text fields in elasticsearch module #10414
  • Parse more fields from elasticsearch audit log #10356
  • Ingest ES structured audit logs #10352
  • Updating Metricbeat stack modules to ECS #10350

Changes in 6.x:

  • Parse more fields from elasticsearch audit log #10385
  • Fixing regression in macOS path #10351

Changes in 6.6:

  • Teach elasticsearch/audit fileset to parse out some more fields #10137
  • Elasticsearch/audit fileset should be more lenient in parsing node name #10135
  • Allow users to convert timezone in elasticsearch module filesets #9761