Metrics

The Java agent tracks certain system and application metrics. Some of them have built-in visualizations and some can only be visualized with custom Kibana dashboards.

These metrics will be sent regularly to the APM Server and from there to Elasticsearch. You can adjust the interval with the setting metrics_interval.

The metrics will be stored in the apm-* index and have the processor.event property set to metric.

Host metrics. As of version 6.6, these metrics will be visualized in the APM app.

For more system metrics, consider installing metricbeat on your hosts.

Linux’s cgroup metrics.

JVM-specific metrics

Java Management Extensions (JMX) provides a common management interface on the JVM and is often used to expose internal metrics through this interface.

The Elastic APM agent is able to connect to the JMX interface directly without needing additional credentials or changing JVM parameters unlike other external tools like VisualVM or Jconsole. Also, it only captures metrics and does not expose the whole JMX management interface to the end-user.

JMX metrics to capture need to be configured through the capture_jmx_metrics option.

To power the Time spent by span type graph, the agent collects summarized metrics about the timings of spans and transactions, broken down by span type.

There are cases where you would want to use the agent only to collect and ship metrics, without tracing any Java code. In such cases, you may set the instrument config option to false. By doing so, the agent will minimize its effect on the application, while still collecting and sending metrics to the APM Server.

The elastic APM Java Agent supports collecting metrics defined via OpenTelemetry. See the corresponding documentation section for details.

The Elastic APM Java agent lets you use the popular metrics collection framework Micrometer to track custom application metrics.

Some use cases for tracking custom metrics from your application include monitoring performance-related things like cache statistics, thread pools, or page hits. However, you can also track business-related metrics such as revenue and correlate them with performance metrics. Metrics registered to a Micrometer MeterRegistry are aggregated in memory and reported every metrics_interval. Based on the metadata about the service and the timestamp, you can correlate metrics with traces. The advantage is that the metrics won’t be affected by the sampling rate and usually take up less space. That is because not every event is stored individually.

The limitation of tracking metrics is that you won’t be able to attribute a value to a specific transaction. If you’d like to do that, add labels to your transaction instead of tracking the metric with Micrometer. The tradeoff here is that you either have to do 100% sampling or account for the missing events. The reason for that is that if you set your sampling rate to 10%, for example, you’ll only be storing one out of 10 requests. The labels you set on non-sampled transactions will be lost.

Attach the agent, and you’re done! The agent automatically detects all MeterRegistry instances and reports all metrics to APM Server (in addition to where they originally report). When attaching the agent after the application has already started, the agent detects a MeterRegistry when calling any public method on it. If you are using multiple registries within a CompoundMeterRegistry, the agent only reports the metrics once.

Use Discover to validate that metrics are successfully reported to Kibana.

You should now see documents containing both metrics collected by the APM agent and custom metrics from Micrometer. Narrow your search with a known Micrometer metric field. For example, if you know you have registered the metric name cache.puts in the Micrometer MeterRegistry, add cache_puts: * (dots are replaced with underscores) to your search to return only Micrometer metrics documents.

TSVB is the recommended visualization for Micrometer metrics. First, make sure to select the right aggregation. The most common options are:

It’s common to group metrics by attributes, including Micrometer labels or attributes already collected by APM agents. This could be service versions, runtime versions, or even cloud metadata.

Declare a dependency to Micrometer:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-core</artifactId>
    <version>${micrometer.version}</version>
</dependency>

Create a Micrometer MeterRegistry.

MeterRegistry registry = new SimpleMeterRegistry(new SimpleConfig() {

        @Override
        public CountingMode mode() {
            // to report the delta since the last report
            // this makes building dashbaords a bit easier
            return CountingMode.STEP;
        }

        @Override
        public Duration step() {
            // the duration should match metrics_interval, which defaults to 30s
            return Duration.ofSeconds(30);
        }

        @Override
        public String get(String key) {
            return null;
        }
    }, Clock.SYSTEM);

The easiest way to get started with Spring Boot is to add a dependency to Spring Boot Actuator. Spring Boot Actuator provides dependency management and auto-configuration for Micrometer.

Use the management.metrics.export.simple prefix to configure via application.properties

management.metrics.export.simple.enabled=true
management.metrics.export.simple.step=30s
management.metrics.export.simple.mode=STEP

This section lists all supported Micrometer Meter s and describes how they are mapped to Elasticsearch documents.

Micrometer tags are nested under labels. Example:

"labels": {
  "tagKey1": "tagLabel1",
  "tagKey2": "tagLabel2",
}

Labels are great for breaking down metrics by different dimensions. Although there is no upper limit, note that a high number of distinct values per label (aka high cardinality) may lead to higher memory usage, higher index sizes, and slower queries. Also, make sure the number of distinct tag keys is limited to avoid mapping explosions.

Depending on the meter type, some meters might be exported as multiple metrics to elasticsearch. The resulting fields are shown below for each meter type. Note that the disable_metrics option operates on the original name of the metric, not on the generated fields. For that reason it is only possible to disable all fields of a given metric, not individual ones.

The agent internally uses a queue to buffer the various events (e.g. transactions, spans, metrics) before sending them to the APM server. When agent_reporter_health_metrics is enabled, the agent will expose several metrics regarding the health state of this queue and the network connectivity to the APM server. In addition, if agent_background_overhead_metrics is enabled, the agent will continuously measure the resource consumption of its own background tasks and provide the results as metrics.