Ingest data from Beats to Elasticsearch Service with Logstash as a proxy

This guide explains how to ingest data from Filebeat and Metricbeat to Logstash as an intermediary, and then send that data to Elasticsearch Service. Using Logstash as a proxy limits your Elastic stack traffic through a single, external-facing firewall exception or rule. Consider the following features of this type of setup:

The configuration in this example makes use of the System module, available for both Filebeat and Metricbeat. Filebeat’s System sends server system log details (that is, login success/failures, sudo superuser do command usage, and other key usage details). Metricbeat’s System module sends memory, CPU, disk, and other server usage metrics.

In the following sections you are going to learn how to:

Time required: 1 hour

Prefer not to subscribe to yet another service? You can also get Elasticsearch Service through AWS, Azure, and GCP marketplaces.

When connecting to Elasticsearch Service you can use a Cloud ID to specify the connection details. You must pass the Cloud ID that you can find in the cloud console. Find your Cloud ID by going to the Kibana main menu and selecting Management > Integrations, and then selecting View deployment details.

To connect to, stream data to, and issue queries with Elasticsearch Service, you need to think about authentication. Two authentication mechanisms are supported, API key and basic authentication. Here, to get you started quickly, we’ll show you how to use basic authentication, but you can also generate API keys as shown later on. API keys are safer and preferred for production environments.

Download and unpack Logstash on the local machine that hosts Beats or another machine granted access to the Beats machines.

Now that Logstash is downloaded and your Elasticsearch Service deployment is set up, you can configure Metricbeat to send operational data to Logstash.

Install Metricbeat as close as possible to the service that you want to monitor. For example, if you have four servers with MySQL running, we recommend that you run Metricbeat on each server. This allows Metricbeat to access your service from localhost. This setup does not cause any additional network traffic and enables Metricbeat to collect metrics even in the event of network problems. Metrics from multiple Metricbeat instances are combined on the Logstash server.

If you have multiple servers with metrics data, repeat the following steps to configure Metricbeat on each server.

Download Metricbeat

Download Metricbeat and unpack it on the local server from which you want to collect data.

About Metricbeat modules

Metricbeat has many modules available that collect common metrics. You can configure additional modules as needed. For this example we’re using Metricbeat’s default configuration, which has the System module enabled. The System module allows you to monitor servers with the default set of metrics: cpu, load, memory, network, process, process_summary, socket_summary, filesystem, fsstat, and uptime.

Load the Metricbeat Kibana dashboards

Metricbeat comes packaged with example dashboards, visualizations, and searches for visualizing Metricbeat data in Kibana. Before you can use the dashboards, you need to create the data view (formerly index pattern) metricbeat-*, and load the dashboards into Kibana. This needs to be done from a local Beats machine that has access to the Elasticsearch Service deployment.

sudo ./metricbeat setup \
  -E cloud.id=<cloudID> \ 
  -E cloud.auth=<username>:<password> 

Your results should be similar to the following:

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

# ---------------- Logstash Output -----------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"] 

The next step is to configure Filebeat to send operational data to Logstash. As with Metricbeat, install Filebeat as close as possible to the service that you want to monitor.

Download Filebeat

Download Filebeat and unpack it on the local server from which you want to collect data.

Enable the Filebeat system module

Filebeat has many modules available that collect common log types. You can configure additional modules as needed. For this example we’re using Filebeat’s System module. This module reads in the various system log files (with information including login successes or failures, sudo command usage, and other key usage details) based on the detected operating system. For this example, a Linux-based OS is used and Filebeat ingests logs from the /var/log/ folder. It’s important to verify that Filebeat is given permission to access your logs folder through standard file and folder permissions.

- module: system
  # Syslog
  syslog:
    enabled: true

  # Authorization logs
  auth:
    enabled: true

From the <localpath>/filebeat-<version> directory, run the filebeat modules command as shown:

./filebeat modules enable system

The system module is now enabled in Filebeat and it will be used the next time Filebeat starts.

Load the Filebeat Kibana dashboards

Filebeat comes packaged with example Kibana dashboards, visualizations, and searches for visualizing Filebeat data in Kibana. Before you can use the dashboards, you need to create the data view filebeat-*, and load the dashboards into Kibana. This needs to be done from a Beats machine that has access to the Internet.

sudo ./filebeat setup \
  -E cloud.id=<cloudID> \ 
  -E cloud.auth=<username>:<password> 

Your results should be similar to the following:

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
Loaded machine learning job configurations
Loaded Ingest pipelines

The data views for filebeat-* and metricbeat-* are now available in Elasticsearch. To verify:

Finish configuring Filebeat

# ---------------- Logstash Output -----------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"] 

Now the Filebeat and Metricbeat are set up, let’s configure a Logstash pipeline to input data from Beats and send results to the standard output. This enables you to verify the data output before sending it for indexing in Elasticsearch.

Now, let’s try out the Logstash pipeline with the Metricbeats and Filebeats configurations from the prior steps. Each Beat sends data into a Logstash pipeline, and the results display on the standard output where you can verify that everything looks correct.

Test Metricbeat to stdout

Test Filebeat to stdout

In this section, you configure Logstash to send the Metricbeat and Filebeat data to Elasticsearch. You modify the beats.conf created earlier, and specify the output credentials needed for our Elasticsearch Service deployment. Then, you start Logstash to send the Beats data into Elasticsearch.

In this section, you log into Elasticsearch Service, open Kibana, and view the Kibana dashboards populated with our Metricbeat and Filebeat data.

View the Metricbeat dashboard

View the Filebeat dashboard

Now, you should have a good understanding of how to configure Logstash to ingest data from multiple Beats. You have the basics needed to begin experimenting with your own combination of Beats and modules.