Journalbeat quick start: installation and configuration

edit

Journalbeat quick start: installation and configuration

edit

This guide describes how to get started quickly with log data collection from systemd journals. You’ll learn how to:

  • install Journalbeat on each system you want to monitor
  • specify the location of your log files
  • parse log data into fields and send it to Elasticsearch
  • visualize the log data in Kibana

Before you begin

edit

You need Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it.

To get started quickly, spin up a deployment of our hosted Elasticsearch Service. The Elasticsearch Service is available on AWS, GCP, and Azure. Try it out for free.

Step 1: Install Journalbeat

edit

Install Journalbeat on all the servers you want to monitor.

To download and install Journalbeat, use the commands that work with your system:

curl -L -O https://artifacts.elastic.co/downloads/beats/journalbeat/journalbeat-7.10.2-amd64.deb
sudo dpkg -i journalbeat-7.10.2-amd64.deb

Other installation options

edit

Step 2: Connect to the Elastic Stack

edit

Connections to Elasticsearch and Kibana are required to set up Journalbeat.

Set the connection information in journalbeat.yml. To locate this configuration file, see Directory layout.

Specify the cloud.id of your Elasticsearch Service, and set cloud.auth to a user who is authorized to set up Journalbeat. For example:

cloud.id: "staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw=="
cloud.auth: "journalbeat_setup:YOUR_PASSWORD" 

This examples shows a hard-coded password, but you should store sensitive values in the secrets keystore.

To learn more about required roles and privileges, see Grant users access to secured resources.

You can send data to other outputs, such as Logstash, but that requires additional configuration and setup.

Step 3: Configure Journalbeat

edit

Before running Journalbeat, specify the location of the systemd journal files and configure how you want the files to be read. If you accept the default configuration, Journalbeat reads from the local journal.

  1. In journalbeat.yml, specify a list of paths to your systemd journal files. Each path can be a directory path (to collect events from all journals in a directory), or a file path. For example:

    journalbeat.inputs:
    - paths:
      - "/dev/log"
      - "/var/log/messages/my-journal-file.journal"
      seek: head

    If no paths are specified, Journalbeat reads from the default journal.

  2. Set the seek option to control the position where Journalbeat starts reading the journal. The available options are head, tail, and cursor. The default is cursor, which means that on first read, Journalbeat starts reading at the beginning of the file, but continues reading at the last known position after a reload or restart. For more detail about the settings, see the reference docs for the seek option.
  3. (Optional) Set the include_matches option to filter entries in journald before collecting any log events. This reduces the number of events that Journalbeat needs to process. For example, to fetch only Redis events from a Docker container tagged as redis, use:

    journalbeat.inputs:
    - paths: []
      include_matches:
        - "CONTAINER_TAG=redis"
        - "_COMM=redis"

To test your configuration file, change to the directory where the Journalbeat binary is installed, and run Journalbeat in the foreground with the following options specified: ./journalbeat test config -e. Make sure your config files are in the path expected by Journalbeat (see Directory layout), or use the -c flag to specify the path to the config file.

For more information about configuring Journalbeat, also see:

Step 4: Set up assets

edit

Journalbeat comes with predefined assets for parsing, indexing, and visualizing your data. To load these assets:

  1. Make sure the user specified in journalbeat.yml is authorized to set up Journalbeat.
  2. From the installation directory, run:

    journalbeat setup -e

    -e is optional and sends output to standard error instead of the configured log output.

This step loads the recommended index template for writing to Elasticsearch.

A connection to Elasticsearch (or Elasticsearch Service) is required to set up the initial environment. If you’re using a different output, such as Logstash, see Load the index template manually.

Step 5: Start Journalbeat

edit

Before starting Journalbeat, modify the user credentials in journalbeat.yml and specify a user who is authorized to publish events.

To start Journalbeat, run:

sudo service journalbeat start

If you use an init.d script to start Journalbeat, you can’t specify command line flags (see Command reference). To specify flags, start Journalbeat in the foreground.

Also see Journalbeat and systemd.

Journalbeat is now ready to send journal events to the Elasticsearch.

Step 6: View your data in Kibana

edit

There is currently no dashboard available for Journalbeat. To start exploring your data, go to the Discover app in Kibana. From there, you can submit search queries, filter the search results, and view document data.

To learn how to build visualizations and dashboards to view your data, see the Kibana User Guide.

What’s next?

edit

Now that you have your logs streaming into Elasticsearch, learn how to unify your logs, metrics, uptime, and application performance data.

  1. Ingest data from other sources by installing and configuring other Elastic Beats:

    Elastic Beats To capture

    Metricbeat

    Infrastructure metrics

    Filebeat

    Logs

    Winlogbeat

    Windows event logs

    Heartbeat

    Uptime information

    APM

    Application performance metrics

    Auditbeat

    Audit events

  2. Use the Observability apps in Kibana to search across all your data:

    Elastic apps Use to

    Metrics app

    Explore metrics about systems and services across your ecosystem

    Logs app

    Tail related log data in real time

    Uptime app

    Monitor availability issues across your apps and services

    APM app

    Monitor application performance

    SIEM app

    Analyze security events

The Logs app shows logs from filebeat-* indices by default. To show Journalbeat indices, configure the source to include journalbeat-*. You can do this in the Logs app when you configure the source, or you can modify the Kibana configuration. For example:

xpack.infra:
 sources:
   default:
     logAlias: "filebeat-*,journalbeat-*"