Ingest metrics with Metricbeat

edit

If you haven’t already, you need to install Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it. For more information, see Spin up the Elastic Stack.

Install and configure Metricbeat on your servers to collect and preprocess system and service metrics, such as information about running processes, as well as CPU, memory, disk, and network utilization numbers.

Metricbeat comes with predefined assets for parsing, indexing, and visualizing your data. To load these assets, Metricbeat uses modules, before sending them to Elasticsearch. Each integration defines the basic logic for collecting data from specific services, such as Redis or MySQL. A module consists of metricsets that fetch and structure the data. To learn more, see How Metricbeat works.

Step 1: Install Metricbeat
edit

Install Metricbeat as close as possible to the service you want to monitor.

To download and install Metricbeat, use the commands that work with your system:

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.3.3-amd64.deb
sudo dpkg -i metricbeat-8.3.3-amd64.deb
Other installation options
edit
Step 2: Connect to Elasticsearch and Kibana
edit

Connections to Elasticsearch and Kibana are required to set up Metricbeat.

Set the connection information in metricbeat.yml. To locate this configuration file, see Directory layout.

Specify the cloud.id of your Elasticsearch Service, and set cloud.auth to a user who is authorized to set up Metricbeat. For example:

cloud.id: "staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw=="
cloud.auth: "metricbeat_setup:YOUR_PASSWORD" 

This examples shows a hard-coded password, but you should store sensitive values in the secrets keystore.

To learn more about required roles and privileges, see Grant users access to secured resources.

Step 3: Enable and configure modules
edit
  1. Identify the modules you need to enable. To see a list of available modules, run:

    metricbeat modules list
  2. From the installation directory, enable one or more modules. If you accept the default configuration without enabling additional modules, Metricbeat collects system metrics only.

    The following command enables the nginx config in the modules.d directory:

    metricbeat modules enable nginx

    See the modules command to learn more about this command. If you are using a Docker image, see Run Metricbeat on Docker.

  3. In the module config under modules.d, change the module settings to match your environment. See Standard config options for more about available settings.

To test your configuration file, change to the directory where the Metricbeat binary is installed, and run Metricbeat in the foreground with the following options specified: ./metricbeat test config -e. Make sure your config files are in the path expected by Metricbeat (see Directory layout), or use the -c flag to specify the path to the config file.

For more information about configuring Metricbeat, also see:

Step 4: Set up assets
edit

Metricbeat comes with predefined assets for parsing, indexing, and visualizing your data. To load these assets:

  1. Make sure the user specified in metricbeat.yml is authorized to set up Metricbeat.
  2. From the installation directory, run:

    metricbeat setup -e

    -e is optional and sends output to standard error instead of the configured log output.

This step loads the recommended index template for writing to Elasticsearch and deploys the sample dashboards for visualizing the data in Kibana.

A connection to Elasticsearch (or Elasticsearch Service) is required to set up the initial environment. If you’re using a different output, such as Logstash, see Load the index template manually and Load Kibana dashboards.

Step 5: Start Metricbeat
edit

Before starting Metricbeat, modify the user credentials in metricbeat.yml and specify a user who is authorized to publish events.

To start Metricbeat, run:

sudo service metricbeat start

If you use an init.d script to start Metricbeat, you can’t specify command line flags (see Command reference). To specify flags, start Metricbeat in the foreground.

Also see Metricbeat and systemd.

Step 6: Confirm metrics are ingested
edit

Let’s confirm your data is correctly ingested to your cluster.

  1. Launch Kibana:

    1. Log in to your Elastic Cloud account.
    2. Navigate to the Kibana endpoint in your deployment.
  2. Open the main menu, then click Discover
  3. Select metricbeat-* as your data view.

    Each document in the index that matches the metricbeat-* data view is displayed. By default, Discover shows data for the last 15 minutes.

Now let’s have a look at the Metrics app.