Configure inputs

edit

By default, Journalbeat reads log events from the default systemd journals. To specify other journal files, set the paths option in the journalbeat.inputs section of the journalbeat.yml file. Each path can be a directory path (to collect events from all journals in a directory), or a file path. For example:

journalbeat.inputs:
- paths:
  - "/dev/log"
  - "/var/log/messages/my-journal-file.journal"

Within the configuration file, you can also specify options that control how Journalbeat reads the journal files and which fields are sent to the configured output. See Configuration options for a list of available options.

The following examples show how to configure Journalbeat for some common use cases.

Example 1: Monitor multiple journals under the same directoryThis example configures Journalbeat to read from multiple journals that are stored under the same directory. Journalbeat merges all journals under the directory into a single event stream and reads the events. With seek set to cursor, Journalbeat starts reading at the beginning of the journal, but will continue reading at the last known position after a reload or restart.

journalbeat.inputs:
- paths: ["/path/to/journal/directory"]
  seek: cursor

Example 2: Fetch log events for Redis running on Docker (uses field names from systemd)This example configures Journalbeat to fetch log events for Redis running in a Docker container. The fields are matched using field names from the systemd journal.

journalbeat.inputs:
- paths: []
  include_matches:
    - "CONTAINER_TAG=redis"
    - "_COMM=redis"

Example 3: Fetch log events for Redis running on Docker (uses translated field names)This example also configures Journalbeat to fetch log events for Redis running in a Docker container. However, in this example the fields are matched using the translated field names provided by Journalbeat.

journalbeat.inputs:
- paths: []
  include_matches:
    - "container.image.tag=redis"
    - "process.name=redis"

Configuration options

edit

You can specify the following options to configure how Journalbeat reads the journal files.

paths

edit

A list of paths that will be crawled and fetched. Each path can be a directory path (to collect events from all journals in a directory), or a file path. If you specify a directory, Journalbeat merges all journals under the directory into a single journal and reads them.

If no paths are specified, Journalbeat reads from the default journal.

backoff

edit

The number of seconds to wait before trying to read again from journals. The default is 1s.

max_backoff

edit

The maximum number of seconds to wait before attempting to read again from journals. The default is 60s.

seek

edit

The position to start reading the journal from. Valid settings are:

  • head: Starts reading at the beginning of the journal. After a restart, Journalbeat resends all log messages in the journal.
  • tail: Starts reading at the end of the journal. After a restart, Journalbeat resends the last message, which might result in duplicates. If multiple log messages are written to a journal while Journalbeat is down, only the last log message is sent on restart.
  • cursor: On first read, starts reading at the beginning of the journal. After a reload or restart, continues reading at the last known position.

When specified under paths, the seek setting applies to all journals under the configured paths. When specified directly under the journalbeat namespace, the setting applies to all journals read by Journalbeat.

If you have old log files and want to skip lines, start Journalbeat with seek: tail specified. Then stop Journalbeat, set seek: cursor, and restart Journalbeat.

include_matches

edit

A list of filter expressions used to match fields. The format of the expression is field=value. Journalbeat fetches all events that exactly match the expressions. Pattern matching is not supported.

To reference fields, use one of the following:

  • The field name used by the systemd journal. For example, CONTAINER_TAG=redis (see a full example).
  • The translated field name used by Journalbeat. For example, container.image.tag=redis (see a full example). Journalbeat does not translate all fields from the journal. For custom fields, use the name specified in the systemd journal.

When specified under paths, the include_matches filter is applied to all journals under the configured paths. When specified directly under the journalbeat namespace, the setting applies to all journals read by Journalbeat.

Translated field names

edit

You can use the following translated names in filter expressions to reference journald fields:

Journald field name

Translated name

COREDUMP_UNIT

journald.coredump.unit

COREDUMP_USER_UNIT

journald.coredump.user_unit

OBJECT_AUDIT_LOGINUID

journald.object.audit.login_uid

OBJECT_AUDIT_SESSION

journald.object.audit.session

OBJECT_CMDLINE

journald.object.cmd

OBJECT_COMM

journald.object.name

OBJECT_EXE

journald.object.executable

OBJECT_GID

journald.object.gid

OBJECT_PID

journald.object.pid

OBJECT_SYSTEMD_OWNER_UID

journald.object.systemd.owner_uid

OBJECT_SYSTEMD_SESSION

journald.object.systemd.session

OBJECT_SYSTEMD_UNIT

journald.object.systemd.unit

OBJECT_SYSTEMD_USER_UNIT

journald.object.systemd.user_unit

OBJECT_UID

journald.object.uid

_AUDIT_LOGINUID

process.audit.login_uid

_AUDIT_SESSION

process.audit.session

_BOOT_ID

host.boot_id

_CAP_EFFECTIVE

process.capabilites

_CMDLINE

process.cmd

_CODE_FILE

journald.code.file

_CODE_FUNC

journald.code.func

_CODE_LINE

journald.code.line

_COMM

process.name

_EXE

process.executable

_GID

process.uid

_HOSTNAME

host.name

_KERNEL_DEVICE

journald.kernel.device

_KERNEL_SUBSYSTEM

journald.kernel.subsystem

_MACHINE_ID

host.id

_MESSAGE

message

_PID

process.pid

_PRIORITY

syslog.priority

_SYSLOG_FACILITY

syslog.facility

_SYSLOG_IDENTIFIER

syslog.identifier

_SYSLOG_PID

syslog.pid

_SYSTEMD_CGROUP

systemd.cgroup

_SYSTEMD_INVOCATION_ID

systemd.invocation_id

_SYSTEMD_OWNER_UID

systemd.owner_uid

_SYSTEMD_SESSION

systemd.session

_SYSTEMD_SLICE

systemd.slice

_SYSTEMD_UNIT

systemd.unit

_SYSTEMD_USER_SLICE

systemd.user_slice

_SYSTEMD_USER_UNIT

systemd.user_unit

_TRANSPORT

systemd.transport

_UDEV_DEVLINK

journald.kernel.device_symlinks

_UDEV_DEVNODE

journald.kernel.device_node_path

_UDEV_SYSNAME

journald.kernel.device_name

_UID

process.uid

The following translated fields for Docker are also available:

CONTAINER_ID

container.id_truncated

CONTAINER_ID_FULL

container.id

CONTAINER_NAME

container.name

CONTAINER_PARTIAL_MESSAGE

container.partial

CONTAINER_TAG

container.image.tag

index

edit

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "journalbeat-myindex-2019.12.13".