Configuring Filebeat

To configure Filebeat, you edit the filebeat.yml file. Here is a sample of filebeat section of the filebeat.yml file:

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      # Paths that should be crawled and fetched. Glob based paths.
      # For each file found under this path, a harvester is started.
      paths:
        - "/var/log/*.log"
      # - c:\programdata\elasticsearch\logs\*

      # Type of the files. Based on this the way the file is read is decided.
      # The different types cannot be mixed in one prospector
      #
      # Possible options are:
      # * log: Reads every line of the log file (default)
      # * stdin: Reads the standard in
      input_type: log

Filebeat uses predefined default values for most configuration options. For the most basic Filebeat configuration, you can define a single prospector with a single path. For example:

filebeat:
  prospectors:
    -
      paths:
        - "/var/log/*.log"

The prospector in this example harvests all files in the path /var/log/*.log, which means that Filebeat will harvest all files in the directory /var/log/ that end with .log. All patterns supported by Golang Glob are also supported here.

To fetch all files from a predefined level of subdirectories, the following pattern can be used: /var/log/*/*.log. This fetches all .log files from the subfolders of /var/log. It does not fetch log files from the /var/log folder itself. Currently it is not possible to recursively fetch all files in all subdirectories of a directory.

A config file can contain multiple prospectors and multiple paths per prospector as shown in the following example.

filebeat:
  prospectors:
    -
      paths:
        - /var/log/system.log
        - /var/log/wifi.log
    -
      paths:
        - "/var/log/apache/*"

The config file in the example starts two prospectors. The first prospector has two harvesters, one harvesting the system.log file, and the other harvesting wifi.log. The second prospector starts a harvester for each file in the apache directory.

See Configuration Options for more details about each configuration option.