Christoph Heger

EDOT SDK central configuration using OpAmp in Elastic Observability

Learn how to configure the Elastic Distributions of OpenTelemetry (EDOT) SDKs centrally via the EDOT Collector using OpAmp in Elastic Observability at scale.

10 min read
OpenTelemetryJavaPythonAPM
EDOT SDK central configuration using OpAmp in Elastic Observability

Managing configuration changes to the large number of services using Elastic Distribution of OpenTelemetry (EDOT) SDKs can be challenging and time-consuming. OpenTelemetry has the Open Agent Management Protocol (OpAMP) which allows Elastic to manage these SDKs through the EDOT collector. Elastic's APM Agents provided this capability in Elastic Observability. Using this functionality together with OpAMP you can now centrally manage your SDKs from Elastic Observability, via the EDOT Collector which uses OpAMP to help manage configuration changes to the multitude of services using the EDOT SDKs.

In this article, we will explore the central configuration capabilities for EDOT SDKs with the EDOT Gateway Collector. You will learn how to configure the EDOT SDKs and the EDOT Gateway Collector to enable central configuration. Finally, we'll cover the configuration settings supported through central configuration.

Central configuration based on the OpenTelemetry Open Agent Management Protocol

The OpenTelemetry project provides OpAMP for the remote management of large fleets of data collection Agents among other capabilities. The central management of EDOT SDKs leverages OpAMP for dispatching the configurations. It's a client-server network protocol where the OpAMP server is part of the EDOT Collector and the OpAMP client is part of the EDOT SDK. The EDOT SDK polls at regular intervals the OpAMP server for configuration updates. The OpAMP server in the EDOT collector is part of the Elastic APM central configuration extension. This extension reads the configuration for the EDOT SDKs from Elasticsearch. The

apmconfigextension
, which is the technical name of the Elastic APM central configuration extension, is included and needs to be configured in the EDOT Collector configuration to be activated. The OpAMP specification includes WebSocket and plain HTTP connection for transport. The EDOT SDKs pull the configuration utilizing the plain HTTP connection for transport from the EDOT Collector.

Prerequisites

Central configuration of EDOT SDKs requires a standalone EDOT Collector running in Gateway mode. Other collectors like the OpenTelemetry contrib collector or a custom distribution of the collector you build yourself require the Elastic APM central configuration extension.

EDOT versions supporting central configuration

The following table gives an overview of the versions of the EDOT SDKs and EDOT Collector that provide central configuration support. Applications and services have to be instrumented with the EDOT SDKs with a version listed below to pull and apply central configuration changes.

EDOTVersion
Android1.2.0+
iOS1.4.0+
Java1.5.0+
Node.js1.2.0+
PHP1.1.1+
Python1.4.0+
Collector (Gateway mode)8.19, 9.1+

Central configuration is not blocking application startup for EDOT Java, Node.js, PHP and Python. The application starts with default configuration or the configuration provided by environment variables. When the central configuration settings are successfully pulled, they will take precedence over local configuration settings. EDOT .NET currently does not support central configuration although it’s planned to add support.

Furthermore, central configuration for EDOT SDKs is not supported on Elastic Cloud Serverless or the Elastic Cloud Managed OTLP Endpoint, yet. The following table gives an overview of the versions of the Elastic stack that support central configuration of EDOT SDKs.

ElasticVersion
Self-managed9.1.0+
Elastic Cloud Hosted9.1.0+

Retrieve Elasticsearch endpoint and API key

The Elastic APM central configuration extension needs the Elasticsearch endpoint in the configuration to be able to read the central configuration settings for the EDOT SDKs. The Elasticsearch endpoint is the same that the Elasticsearch exporter uses to export telemetry data to Elasticsearch. The central configuration documentation describes the steps to obtain the endpoint and the API key in more detail.

Enable central configuration for EDOT

To enable central configuration, the EDOT Collector needs the

apmconfigextension
to be configured as part of the
extensions
section. This requires the Elasticsearch endpoint obtained above and an Elasticsearch API key. The environment variable
ELASTIC_OTEL_OPAMP_ENDPOINT
needs to be set to enable central configuration in the EDOT SDK.

In the following, the configuration is explained for the EDOT Gateway Collector and the EDOT SDKs.

Configure the EDOT Collector to enable central configuration

Central configuration support in the EDOT Collector is enabled by adding the configuration of the Elastic APM central configuration extension configuration to the configuration file. For the authentication of the

apmconfigextension
with the Elasticsearch endpoint,
bearertokenauth
authenticator is configured. This configures a client type authenticator for outgoing requests to the Elasticsearch endpoint. The
apmconfigextension
acts as client and Elasticsearch endpoint as server. The
apmconfig
section configures the OpAMP server endpoint. EDOT SDKs will connect to the endpoint to fetch the configuration. The
service
section activates the
apmconfig
and
bearertokenauth
extension. The following code snippet shows the configuration excerpt of the EDOT Collector including the
bearertoken
authenticator configuration and the
apmconfig
configuration for central configuration.

extensions:
  bearertokenauth:
    scheme: "APIKey"
    token: "<ENCODED_ELASTICSEARCH_APIKEY>"
  source:
     elasticsearch:
       endpoint: "<YOUR_ELASTICSEARCH_ENDPOINT>"
       auth:
         authenticator: bearertokenauth
  apmconfig:
    opamp:
      protocols:
        http:
          # Default is localhost:4320
          # To specify a custom endpoint, uncomment the following line
          # and set the endpoint to the custom endpoint
          # endpoint: "<CUSTOM_OPAMP_ENDPOINT>"
  service:
    extensions: [bearertokenauth, apmconfig]

Download the EDOT collector and include the configuration from the snippet above in the

otel.yml
configuration file to enable central configuration. The
otel.yml
configuration file examples are available in the EDOT Collector documentation. Consider the example for direct ingestion into Elasticsearch.

Configure the EDOT SDKs to enable central configuration

To enable central configuration in the EDOT SDKs Java, Node.js, PHP and Python, set the

ELASTIC_OTEL_OPAMP_ENDPOINT
environment variable to the OpAMP server endpoint of the EDOT Collector and set the required resource attributes.

Enable central configuration of EDOT SDKs

The following code snippet shows how to set the

ELASTIC_OTEL_OPAMP_ENDPOINT
environment variable with the
export
command in a shell.

export ELASTIC_OTEL_OPAMP_ENDPOINT="http://<your-opamp-end-point>:4320/v1/opamp"

<your-opamp-end-point>
must be set to the address or host name of the EDOT Gateway Collector that provides the OpAMP server endpoint for central configuration.

When you are using EDOT SDKs for mobile, the documentation shows how to activate central configuration support in EDOT Android and EDOT iOS.

Configure resource attributes

Central configuration requires the OpenTelemetry resource attributes

service.name
and
deployment.environment.name
to be set. While
service.name
is mandatory,
deployment.environment.name
is optional but recommended. If
deployment.environment.name
is unset, no configuration can be created that applies to a whole environment.

Set the

OTEL_RESOURCE_ATTRIBUTES
environment variable including the
service.name
and
deployment.environment.name
like in the following code snippet. The key-value pairs are concatenated with a comma as separator and provided as value to the
OTEL_RESOURCE_ATTRIBUTES
environment variable.

export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.name=my-app"

Supported configuration settings

The following tables give an overview of the supported central configuration settings at the time of writing.

Non-mobile EDOT SDKs

The table shows the supported central configuration settings of EDOT Java, Node.js, PHP, and Python SDK.

SettingDescriptionJavaNode.jsPHPPythonKibana
logging_level
The EDOT SDKs own logging level1.5.0+1.2.0+1.1.0+1.4.0+9.1.0+
deactivate_instrumentations
Turn off selected instrumentations1.5.0+1.2.0+--9.1.0+
deactivate_all_instrumentations
Turn off all instrumentations1.5.0+1.2.0+--9.1.0+
send_traces
Controls if traces should be sent1.5.0+1.3.0+--9.1.0+
send_metrics
Controls if metrics should be sent1.5.0+1.3.0+--9.1.0+
send_logs
Controls if logs should be sent1.5.0+1.3.0+--9.1.0+
opamp_polling_interval
Time between consecutive central configuration pull requests1.6.0+---9.2.0+ (planned)
sampling_rate
Trace sampling rate for head-based sampling1.6.0+--1.7.0+9.2.0+ (planned)
infer_spans
Activates/Deactivates inferred spans1.7.0+---9.2.0+ (planned)

Mobile EDOT SDKs

The table below shows the supported central configuration settings of EDOT Android, and iOS SDK.

SettingDescriptionAndroidiOSKibana
recording
Record and send telemetry1.2.0+1.4.0+9.1.0+
session_sample_rate
Sampling rate for session-based sampling1.2.0+1.4.0+9.1.0+

Use Elastic Observability to change configuration settings of EDOT SDKs

An application must produce and send telemetry data otherwise the EDOT SDK will not appear in the Agent configuration UI in Elastic Observability. Because the Agent configuration in Elastic Observability has no knowledge about the existence of the EDOT SDK until it's receiving telemetry data. The OpenTelemetry resource attribute

service.name
is used as the key to assign a configuration to an EDOT SDK. Currently, EDOT SDKs will not show up in the Agent Explorer.

Go to Kibana -> Observability -> Applications -> Service Inventory -> Settings -> Agent Configuration to create a new configuration for your EDOT SDK.

The EDOT Java SDK configuration above deactivates all instrumentations by setting

deactivate_all_instrumentations
to
true
. This is useful in situations when switching off the instrumentations is necessary. The
sampling_rate
setting becomes handy when the sampling rate of the EDOT SDK should be changed. A sampling rate can be selected according to the needs as shown below.

Disable central configuration

Disabling central configuration is easy. Remove the environment variable

ELASTIC_OTEL_OPAMP_ENDPOINT
and restart the application to disable central configuration support in the EDOT SDKs. Remove the
apmconfig
extension from the
service
section of the EDOT Collector configuration and restart the collector to disable the Elastic APM central configuration extension in the EDOT Collector.

Elastic's contribution to OpenTelemetry

Elastic is committed to the OpenTelemetry project. Elastic contributed the Java OpAMP client implementation to the OpenTelemetry project (Github PR), and is working on the contribution for Python (Github PR) and Node.js. The OpAMP client for PHP will be part of a larger contribution (Github issue) that Elastic is doing.

Conclusion

In this article, you learned how to configure the EDOT SDKs centrally in Elastic Observability with the Gateway Collector and OpAMP at scale. You learned how to configure the

apmconfigextension
in the collector and how to set the
ELASTIC_OTEL_OPAMP_ENDPOINT
environment variable to enable central configuration in the EDOT SDKs, which versions of the SDKs and collector support central configuration, and what configuration settings are currently supported. Now you can leverage central configuration in large deployments to manage the configuration of the EDOT SDKs.

Share this article

XFacebookLinkedInReddit