Profiler Auto instrumentation

edit

Quick start

edit

The agent can automatically instrument .NET Framework, .NET Core, and .NET applications using the .NET CLR Profiling APIs. The Profiling APIs provide a way to instrument an application or dependency code without code changes.

This approach works with the following

Operating system

Architecture

Windows

Linux **

x64

.NET Framework 4.6.2+*

.NET Core 2.1+

.NET 5+

.NET Core 2.1+

.NET 5+

* Due to binding issues introduced by Microsoft, we recommend at least .NET Framework 4.7.2 for best compatibility.

** Minimum GLIBC version 2.14.

The profiler-based agent only supports 64-bit applications. 32-bit applications aren’t supported.

The profiler-based agent does not currently support ARM.

It instruments the following assemblies:

Integration name NuGet package version(s) Assembly version(s)

AdoNet

part of .NET

System.Data 4.0.0 - 4.*.*

part of .NET

System.Data.Common 4.0.0 - 5.*.*

AspNet

part of .NET Framework

System.Web 4.0.0 - 4.*.*

Kafka

Confluent.Kafka 1.4.0 - 1.*.*

Confluent.Kafka 1.4.0 - 1.*.*

MySqlCommand

MySql.Data 6.7.0 - 8.*.*

MySql.Data 6.7.0 - 8.*.*

NpgsqlCommand

Npgsql 4.0.0 - 7.*.*

Npgsql 4.0.0 - 7.*.*

OracleCommand

Oracle.ManagedDataAccess 12.2.1100 - 21.*.*

Oracle.ManagedDataAccess 4.122.0 - 4.122.*

Oracle.ManagedDataAccess.Core 2.0.0 - 3.*.*

Oracle.ManagedDataAccess 2.0.0 - 3.*.*

RabbitMQ

RabbitMQ.Client 3.6.9 - 6.*.*

RabbitMQ.Client 3.6.9 - 6.*.*

SqlCommand

part of .NET

System.Data 4.0.0 - 4.*.*

System.Data.SqlClient 4.0.0 - 4.*.*

System.Data.SqlClient 4.0.0 - 4.*.*

Microsoft.Data.SqlClient 1.0.0 - 5.*.*

Microsoft.Data.SqlClient 1.0.0 - 5.*.*

SqliteCommand

Microsoft.Data.Sqlite 2.0.0 - 7.*.*

Microsoft.Data.Sqlite 2.0.0 - 7.*.*

System.Data.SQLite 1.0.0 - 2.*.*

System.Data.SQLite 1.0.0 - 2.*.*

The .NET CLR Profiling API allows only one profiler to be attached to a .NET process. In light of this limitation, only one solution that uses the .NET CLR profiling API should be used by an application.

Auto instrumentation using the .NET CLR Profiling API can be used in conjunction with

  • the Public API to perform manual instrumentation.
  • NuGet packages that perform instrumentation using a IDiagnosticsSubscriber to subscribe to diagnostic events.

The version number of NuGet packages referenced by a project instrumented with a profiler must be the same as the version number of profiler zip file used.

General steps

edit

The general steps in configuring profiler auto instrumentation are as follows; See Instrumenting containers and services for configuration for common deployment environments.

  1. Download the elastic_apm_profiler_<version>.zip file from the Releases page of the .NET APM Agent GitHub repository, where <version> is the version number to download. You can find the file under Assets.
  2. Unzip the zip file into a folder on the host that is hosting the application to instrument.
  3. Configure the following environment variables:

    .NET Framework.

    set COR_ENABLE_PROFILING = "1"
    set COR_PROFILER = "{FA65FE15-F085-4681-9B20-95E04F6C03CC}"
    set COR_PROFILER_PATH = "<unzipped directory>\elastic_apm_profiler.dll" 
    set ELASTIC_APM_PROFILER_HOME = "<unzipped directory>"
    set ELASTIC_APM_PROFILER_INTEGRATIONS = "<unzipped directory>\integrations.yml"
    set ELASTIC_APM_SERVER_URL = "<apm server url>" 
    set ELASTIC_APM_SECRET_TOKEN = "<secret token>" 

    <unzipped directory> is the directory to which the zip file was unzipped in step 2.

    The URL of the APM server intake to which traces and metrics should be sent.

    The secret token used by the APM Agent to authenticate with APM server.

    .NET Core / .NET 5+ on Windows.

    set CORECLR_ENABLE_PROFILING = "1"
    set CORECLR_PROFILER = "{FA65FE15-F085-4681-9B20-95E04F6C03CC}"
    set CORECLR_PROFILER_PATH = "<unzipped directory>\elastic_apm_profiler.dll" 
    set ELASTIC_APM_PROFILER_HOME = "<unzipped directory>"
    set ELASTIC_APM_PROFILER_INTEGRATIONS = "<unzipped directory>\integrations.yml"
    set ELASTIC_APM_SERVER_URL = "<apm server url>" 
    set ELASTIC_APM_SECRET_TOKEN = "<secret token>" 

    <unzipped directory> is the directory to which the zip file was unzipped in step 2.

    The URL of the APM server intake to which traces and metrics should be sent.

    The secret token used by the APM Agent to authenticate with APM server.

    .NET Core / .NET 5+ on Linux.

    export CORECLR_ENABLE_PROFILING=1
    export CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}
    export CORECLR_PROFILER_PATH="<unzipped directory>/libelastic_apm_profiler.so" 
    export ELASTIC_APM_PROFILER_HOME="<unzipped directory>"
    export ELASTIC_APM_PROFILER_INTEGRATIONS="<unzipped directory>/integrations.yml"
    export ELASTIC_APM_SERVER_URL = "<apm server url>" 
    export ELASTIC_APM_SECRET_TOKEN = "<secret token>" 

    <unzipped directory> is the directory to which the zip file was unzipped in step 2.

    The URL of the APM server intake to which traces and metrics should be sent.

    The secret token used by the APM Agent to authenticate with APM server.

Ensure you start your application in a context where the set environment variables are visible.

With this setup, the .NET runtime loads Elastic’s CLR profiler into the .NET process, which loads and instantiates the APM agent early in application startup. The profiler monitors methods of interest and injects code to instrument their execution.

Instrumenting containers and services

edit

Using global environment variables causes profiler auto instrumentation to be loaded for any .NET process started on the host. Often, the environment variables should be set only for specific services or containers. The following sections demonstrate how to configure common containers and services.

Docker containers

edit

A build image containing the files for profiler auto instrumentation can be used as part of a multi-stage build

ARG AGENT_VERSION=<VERSION> 

FROM alpine:latest AS build
ARG AGENT_VERSION
WORKDIR /source

# install unzip
RUN apk update && apk add zip curl

# pull down the zip file based on ${AGENT_VERSION} ARG and unzip
RUN curl -L -o elastic_apm_profiler_${AGENT_VERSION}.zip https://github.com/elastic/apm-agent-dotnet/releases/download/v${AGENT_VERSION}/elastic_apm_profiler_${AGENT_VERSION}.zip && \
    unzip elastic_apm_profiler_${AGENT_VERSION}.zip -d /elastic_apm_profiler_${AGENT_VERSION}

Replace <VERSION> with the newest release version number for the profiler zip file to be downloaded (e.g. ARG AGENT_VERSION=1.26.0).

The files can then be copied into a subsequent stage

COPY --from=build /elastic_apm_profiler_${AGENT_VERSION} /elastic_apm_profiler

Environment variables can be added to a Dockerfile to configure profiler auto instrumentation

ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}
ENV CORECLR_PROFILER_PATH=/elastic_apm_profiler/libelastic_apm_profiler.so
ENV ELASTIC_APM_PROFILER_HOME=/elastic_apm_profiler
ENV ELASTIC_APM_PROFILER_INTEGRATIONS=/elastic_apm_profiler/integrations.yml

ENTRYPOINT ["dotnet", "your-application.dll"]

You should also consider how to securely provide the the APM server URL and secret token as environment variables when you run your container. It is not recommended to include the secret token in the image as it may be accidently exposed.

Windows services

edit

Environment variables can be added to specific Windows services by adding an entry to the Windows registry. Using PowerShell

.NET Framework service.

$environment = [string[]]@(
  "COR_ENABLE_PROFILING=1",
  "COR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}",
  "COR_PROFILER_PATH=<unzipped directory>\elastic_apm_profiler.dll",
  "ELASTIC_APM_PROFILER_HOME=<unzipped directory>",
  "ELASTIC_APM_PROFILER_INTEGRATIONS=<unzipped directory>\integrations.yml"
  "ELASTIC_APM_SERVER_URL=<apm server url>"
  "ELASTIC_APM_SECRET_TOKEN=<secret token>")

Set-ItemProperty HKLM:SYSTEM\CurrentControlSet\Services\<service-name> -Name Environment -Value $environment

.NET Core service.

$environment = [string[]]@(
  "CORECLR_ENABLE_PROFILING=1",
  "CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}",
  "CORECLR_PROFILER_PATH=<unzipped directory>\elastic_apm_profiler.dll", 
  "ELASTIC_APM_PROFILER_HOME=<unzipped directory>",
  "ELASTIC_APM_PROFILER_INTEGRATIONS=<unzipped directory>\integrations.yml"
  "ELASTIC_APM_SERVER_URL=<apm server url>" 
  "ELASTIC_APM_SECRET_TOKEN=<secret token>") 

Set-ItemProperty HKLM:SYSTEM\CurrentControlSet\Services\<service-name> -Name Environment -Value $environment 

<unzipped directory> is the directory to which the zip file was unzipped.

The URL of the APM server intake to which traces and metrics should be sent.

The secret token used by the APM Agent to authenticate with APM server.

<service-name> is the name of the Windows service.

The service must then be restarted for the change to take effect. With PowerShell

Restart-Service <service-name>

Internet Information Services (IIS)

edit

For IIS versions before IIS 10, it is not possible to set environment variables scoped to a specific application pool, so environment variables need to set globally.

For IIS 10 onwards, environment variables can be set for an application pool using AppCmd.exe. With PowerShell

.NET Framework.

$appcmd = "$($env:systemroot)\system32\inetsrv\AppCmd.exe"
$appPool = "<application-pool>" 
$profilerHomeDir = "<unzipped directory>" 
$environment = @{
  COR_ENABLE_PROFILING = "1"
  COR_PROFILER = "{FA65FE15-F085-4681-9B20-95E04F6C03CC}"
  COR_PROFILER_PATH = "$profilerHomeDir\elastic_apm_profiler.dll"
  ELASTIC_APM_PROFILER_HOME = "$profilerHomeDir"
  ELASTIC_APM_PROFILER_INTEGRATIONS = "$profilerHomeDir\integrations.yml"
  COMPlus_LoaderOptimization = "1" 
  ELASTIC_APM_SERVER_URL = "<apm server url>" 
  ELASTIC_APM_SECRET_TOKEN = "<secret token>" 
}

$environment.Keys | ForEach-Object {
  & $appcmd set config -section:system.applicationHost/applicationPools /+"[name='$appPool'].environmentVariables.[name='$_',value='$($environment[$_])']"
}

<application-pool> is the name of the Application Pool your application uses. For example, IIS APPPOOL\DefaultAppPool

<unzipped directory> is the full path to the directory in which the zip file was unzipped

Forces assemblies not to be loaded domain-neutral. There is currently a limitation where Profiler auto-instrumentation cannot instrument assemblies when they are loaded domain-neutral. This limitation is expected to be removed in future, but for now, can be worked around by setting this environment variable. See the Microsoft documentation for further details.

The URL of the APM server intake to which traces and metrics should be sent.

The secret token used by the APM Agent to authenticate with APM server.

.NET Core.

$appcmd = "$($env:systemroot)\system32\inetsrv\AppCmd.exe"
$appPool = "<application-pool>" 
$profilerHomeDir = "<unzipped directory>" 
$environment = @{
  CORECLR_ENABLE_PROFILING = "1"
  CORECLR_PROFILER = "{FA65FE15-F085-4681-9B20-95E04F6C03CC}"
  CORECLR_PROFILER_PATH = "$profilerHomeDir\elastic_apm_profiler.dll"
  ELASTIC_APM_PROFILER_HOME = "$profilerHomeDir"
  ELASTIC_APM_PROFILER_INTEGRATIONS = "$profilerHomeDir\integrations.yml"
  ELASTIC_APM_SERVER_URL = "<apm server url>" 
  ELASTIC_APM_SECRET_TOKEN = "<secret token>" 
}

$environment.Keys | ForEach-Object {
  & $appcmd set config -section:system.applicationHost/applicationPools /+"[name='$appPool'].environmentVariables.[name='$_',value='$($environment[$_])']"
}

<application-pool> is the name of the Application Pool your application uses. For example, IIS APPPOOL\DefaultAppPool.

<unzipped directory> is the full path to the directory in which the zip file was unzipped.

The URL of the APM server intake to which traces and metrics should be sent.

The secret token used by the APM Agent to authenticate with APM server.

Ensure that the location of the <unzipped directory> is accessible and executable to the Identity account under which the Application Pool runs.

Once environment variables have been set, stop and start IIS so that applications hosted in IIS will see the new environment variables.

net stop /y was
net start w3svc

systemd / systemctl

edit

Environment variables can be added to specific services run with systemd by creating an environment.env file containing the following

CORECLR_ENABLE_PROFILING=1
CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}
CORECLR_PROFILER_PATH=/<unzipped directory>/libelastic_apm_profiler.so 
ELASTIC_APM_PROFILER_HOME=/<unzipped directory>
ELASTIC_APM_PROFILER_INTEGRATIONS=/<unzipped directory>/integrations.yml
ELASTIC_APM_SERVER_URL=<apm server url> 
ELASTIC_APM_SECRET_TOKEN=<secret token> 

<unzipped directory> is the directory to which the zip file was unzipped.

The URL of the APM server intake to which traces and metrics should be sent.

The secret token used by the APM Agent to authenticate with APM server.

Then adding an EnvironmentFile entry to the service’s configuration file that references the path to the environment.env file

[Service]
EnvironmentFile=/path/to/environment.env
ExecStart=<command> 

the command that starts your service

After adding the EnvironmentFile entry, restart the service.

systemctl reload-or-restart <service>

Profiler environment variables

edit

The profiler auto instrumentation has its own set of environment variables to manage the instrumentation. These are used in addition to agent configuration through environment variables.

ELASTIC_APM_PROFILER_HOME

The home directory of the profiler auto instrumentation. The home directory typically contains:

  • platform specific profiler assemblies
  • a directory for each compatible target framework, where each directory contains supporting managed assemblies for auto instrumentation.
  • an integrations.yml file that determines which methods to target for auto instrumentation
ELASTIC_APM_PROFILER_INTEGRATIONS (optional)
The path to the integrations.yml file that determines which methods to target for auto instrumentation. If not specified, the profiler will assume an integrations.yml exists in the home directory specified by ELASTIC_APM_PROFILER_HOME environment variable.
ELASTIC_APM_PROFILER_EXCLUDE_INTEGRATIONS (optional)
A semi-colon separated list of integrations to exclude from auto-instrumentation. Valid values are those defined in the Integration name column in the integrations table above.
ELASTIC_APM_PROFILER_EXCLUDE_PROCESSES (optional)
A semi-colon separated list of process names to exclude from auto-instrumentation. For example, dotnet.exe;powershell.exe. Can be used in scenarios where profiler environment variables have a global scope that would end up auto-instrumenting applications that should not be.

The following processes are *always* excluded from profiling by default.

  • powershell.exe
  • ServerManager.exe
  • ReportingServicesService.exe
  • RSHostingService.exe
  • RSMananagement.exe
  • RSPortal.exe
  • RSConfigTool.exe

    ELASTIC_APM_PROFILER_EXCLUDE_SERVICE_NAMES (optional)
    A semi-colon separated list of APM service names to exclude from auto-instrumentation. Values defined are checked against the value of ELASTIC_APM_SERVICE_NAME environment variable.

The following service names are *always* excluded from profiling by default.

  • SQLServerReportingServices

    ELASTIC_OTEL_LOG_LEVEL (optional)
    The log level at which the profiler should log. Valid values are
  • trace
  • debug
  • info
  • warn
  • error
  • none

The default value is warn. More verbose log levels like trace and debug can affect the runtime performance of profiler auto instrumentation, so are recommended only for diagnostics purposes.

This takes precedence over the now deprecated ELASTIC_APM_PROFILER_LOG

ELASTIC_OTEL_LOG_DIRECTORY (optional)

The directory in which to write profiler log files. If unset, defaults to

  • %PROGRAMDATA%\elastic\apm-agent-dotnet\logs on Windows
  • /var/log/elastic/apm-agent-dotnet on Linux

If the default directory cannot be written to for some reason, the profiler will try to write log files to a logs directory in the home directory specified by ELASTIC_APM_PROFILER_HOME environment variable.

This takes precedence over the now deprecated ELASTIC_APM_PROFILER_LOG_DIR

The user account under which the profiler process runs must have permission to write to the destination log directory. Specifically, ensure that when running on IIS, the AppPool identity has write permissions in the target directory.

ELASTIC_OTEL_LOG_TARGETS (optional)

A semi-colon separated list of targets for profiler logs. Valid values are

  • file
  • stdout

The default value is file, which logs to the directory specified by ELASTIC_APM_PROFILER_LOG_DIR environment variable.

This takes precedence over the now deprecated ELASTIC_APM_PROFILER_LOG_TARGETS

Troubleshooting

edit
DLLs are blocked on Windows
edit

Windows may automatically block downloaded DLL files if it considers them suspicious.

To unblock a DLL file on Windows, you can do the following:

  • Right-click the DLL file in File Explorer
  • Select Properties
  • In the General tab, look for the Security section at the bottom
  • Select the Unblock check box and click OK
Unblock DLL in Windows file properties