Get up and running
editGet up and running
editThis functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
To get up and running with infrastructure monitoring, you need:
-
An Elasticsearch cluster and Kibana (version 6.5 or later) with a basic license. To learn how to get started quickly, see Getting started with the Elastic Stack.
You can skip having to install Elasticsearch and Kibana by using our hosted Elasticsearch Service on Elastic Cloud. The Elasticsearch Service is available on both AWS and GCP. Try out the Elasticsearch Service for free.
- Beats shippers (version 6.5 or later) installed on each system you want to monitor
You might also need to modify UI settings in Kibana to change default behaviors, such as the index pattern used to query the data, and the timestamp field used for sorting. For more information, see Infrastructure UI Settings and Logs UI Settings.
Install Beats shippers
editTo populate the Infrastructure UI with metrics and log data, you need to install and configure the following shippers:
- Metricbeat for metrics
- Filebeat for log data
To learn how to install and configure the shippers, do one of the following:
-
Follow the instructions in the Add Data section of the Kibana home page. Click Add log data or Add metrics, and follow the links for the types of data you want to collect.
-
If your data source isn’t in the list, or you want to install Beats the old fashioned way:
- Follow the Metricbeat getting started and enable modules for the metrics you want to collect.
- Follow the Filebeat modules quick start and enable modules for the logs you want to collect. If there is no module for the logs you want to collect, see the Filebeat getting started to learn how to configure inputs.
For either approach, you need to enable modules in Filebeat and Metricbeat to populate the Infrastructure UI with data.
Which modules and configuration options do I enable?
editTo populate the Hosts view and add logs, enable:
-
Metricbeat
systemmodule (enabled by default) -
Filebeat
systemmodule -
Other Filebeat modules needed for
your environment, such as
apache2,redis, and so on -
Metricbeat
add_host_metadataprocessor (enabled by default) -
Metricbeat
add_cloud_metadataprocessor (enabled by default)
To populate the Docker view and add logs, enable:
To populate the Kubernetes view and add logs, enable:
Which fields are used for the metrics on the Infrastructue home page?
editThe metrics listed below are provided by the Beats Shippers. Each system type requires their corresponding identity field to be in the same event document:
-
Hosts require
host.name -
Docker containers require
container.id -
Kibernetes pods require
kibernetes.pod.uid
Host Metrics
edit- CPU Usage
-
Average of
system.cpu.user.pctadded to the average ofsystem.cpu.system.pctdivided bysystem.cpu.cores - Memory Usage
-
Average of
system.memory.actual.used.pct - Load
-
Average of
system.load.5 - Inbound Traffic
-
Derivative of the max of
system.netowrk.in.bytesscaled to a 1 second rate - Outbound Traffic
-
Derivative of the max of
system.netowrk.out.bytesscaled to a 1 second rate - Log Rate
- Derivative of the cumulative sum of the document count scaled to a 1 second rate. This metric relies on the same indices as the logs.
Docker Container Metrics
edit- CPU Usage
-
Average of
docker.cpu.total.pct - Memory Usage
-
Average of
docker.memory.usage.pct - Inbound Traffic
-
Derivative of the max of
docker.network.in.bytesscaled to a 1 second rate - Outbound Traffic
-
Derivative of the max of
docker.network.out.bytesscaled to a 1 second rate
Kubernetes Pod Metrics
edit- CPU Usage
-
Average of
kubernetes.pod.cpu.usage.node.pct - Memory Usage
-
Average of
kubernetes.pod.memory.usage.node.pct - Inbound Traffic
-
Derivative of the max of
kubernetes.pod.network.rx.bytesscaled to a 1 second rate - Outbound Traffic
-
Derivative of the max of
kubernetes.pod.network.tx.bytesscaled to a 1 second rate
More about container monitoring
editIf you’re monitoring containers, you can use autodiscover to automatically apply configuration changes in response to changes in your containers. To learn how, see:
Known Workarounds
edit- Running Metricbeat ⇐6.5 with Kibana 6.6+
-
For Kubernetes, you will need to change the "Pod ID" field in the "Configure Source" panel (
xpack.infra.sources.default.fields.podinconfig/kibana.ymlfor Kibana 6.6) tokubernetes.pod.name. There is a caveat for this workaround; if you have two pods with the same name, only one will be visible in the UI.