Loading

Elastic APM connector

Serverless Observability Stack EDOT Collector

The Elastic APM connector generates pre-aggregated metrics from OpenTelemetry trace data. These metrics enable key APM features like service maps, transaction histograms, and service-level indicators with fast query performance. Instead of calculating metrics on the fly from potentially millions of transactions, the APM UIs can query pre-computed metric documents for quick data visualization.

The connector works together with the Elastic APM processor, which enriches OpenTelemetry traces with Elastic-specific attributes to ensure optimal compatibility with Elastic APM UIs.

The elasticapm connector is included by default in EDOT Collector deployments that ingest trace data directly into Elasticsearch. It's not needed when using the Elastic Cloud Managed OTLP Endpoint, as the metric aggregation happens server-side.

In standalone deployments, the Elastic APM connector is used in both agent and gateway modes:

  • Agent mode: The connector is part of the default application and traces collection pipeline. It receives trace data from the pipeline, generates metrics, and forwards them to Elasticsearch.

  • Gateway mode: The connector is part of the Gateway mode pipeline, where it generates metrics from traces received from other collectors running in agent mode before ingesting them into Elasticsearch.

In Kubernetes, the Elastic APM connector runs in the Gateway collectors pipeline when using direct ingestion to Elasticsearch. The Gateway receives traces from DaemonSet collectors, generates APM metrics, and writes both metrics and traces to Elasticsearch.

For more details about the Kubernetes configuration, refer to Default configuration (Kubernetes).

The Elastic APM connector typically requires minimal configuration. Usually, an empty configuration block is sufficient:

connectors:
  elasticapm: {}

When combined with the elasticapm processor in a complete pipeline:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

connectors:
  elasticapm: {}

processors:
  batch:
    send_batch_size: 1000
    timeout: 1s
  elasticapm: {}

exporters:
  elasticsearch/otel:
    endpoints:
      - ${ELASTIC_ENDPOINT}
    api_key: ${ELASTIC_API_KEY}
    mapping:
      mode: otel

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, elasticapm]
      exporters: [elasticapm, elasticsearch/otel]

    metrics/aggregated-otel-metrics:
      receivers: [elasticapm]
      processors: []
      exporters: [elasticsearch/otel]

In this configuration, the elasticapm connector appears as both an exporter in the traces pipeline to generate APM metrics and as a receiver in the metrics pipeline to forward those metrics to Elasticsearch.

The Elastic APM connector generates the following types of aggregated metrics from trace data:

Metric Type Description
Transaction metrics Aggregated statistics about service transactions, including throughput, latency distributions, and success rates. These metrics power service overview pages and transaction group views.
Service destination metrics Metrics that track dependencies between services, showing how services communicate with databases, message queues, and other external systems. These metrics are used to build service maps.
Span metrics Detailed metrics about individual span operations, including database queries, HTTP calls, and other operations. These provide granular performance insights for specific operations.
Service summary metrics High-level service health metrics including error rates, throughput, and overall latency. These metrics enable quick health checks and alerting.

All metrics are generated with appropriate aggregation periods and follow Elastic's metric schema for seamless integration with APM UIs.

The connector aggregates metrics over multiple time intervals to provide different granularities for analysis. By default, metrics are aggregated at three intervals: 1 minute, 10 minutes, and 60 minutes.

Follow these recommendations when using the Elastic APM connector:

  • Always pair with the elasticapm processor: The connector and processor work together to provide the full Elastic APM experience. The processor enriches traces while the connector generates APM metrics. Include both in your pipeline configuration for complete functionality.

  • Place the connector as an exporter in the traces pipeline: Configure the Elastic APM connector as an exporter in your traces pipeline, alongside your final data destination. This ensures the connector receives processed trace data and can generate accurate metrics.

  • Create a separate metrics pipeline for connector output: Set up a dedicated metrics pipeline with the connector as the receiver. This isolates metric handling and makes it easier to apply metric-specific processing if needed.

  • Use only for direct Elasticsearch ingestion: If you're using the Elastic Cloud Managed OTLP Endpoint, you don't need the Elastic APM connector, because the endpoint handles metric aggregation automatically. Using both can cause conflicts or duplicate metrics.

  • Keep the connector updated: The Elastic APM connector evolves with new Elastic APM features. Keep your EDOT Collector version current to benefit from the latest enhancements and compatibility improvements.

  • Monitor connector performance: The connector aggregates metrics in memory before flushing. For high-throughput environments, monitor memory usage and adjust the aggregation interval or deployment resources as needed.

Be aware of these constraints and behaviors when using the Elastic APM connector:

  • Required for Elastic APM metrics: Without the Elastic APM connector, you'll only have raw trace data in Elasticsearch. Service maps, transaction histograms, and other metric-driven APM features might require the pre-aggregated metrics that this connector generates.

  • Not available in contrib OTel Collector: The Elastic APM connector is an Elastic-specific component not included in the standard OpenTelemetry Collector or Collector Contrib distributions. To use it, you must either use EDOT Collector or build a custom collector that includes Elastic's components.

  • Memory usage scales with cardinality: The connector maintains in-memory aggregations for unique combinations of service names, transaction names, and other dimensions. High-cardinality data (many unique values) increases memory requirements. Monitor memory usage in high-cardinality environments.

  • Minimal configuration options: Unlike some connectors, the Elastic APM connector operates with mostly fixed behavior and offers few configuration parameters. While this simplifies setup, it also means you have limited ability to customize the aggregation logic.