« Span compression Transaction sampling »

› › › › ›

Distributed tracing

A trace is a group of transactions and spans with a common root. Each trace tracks the entirety of a single request. When a trace travels through multiple services, as is common in a microservice architecture, it is known as a distributed trace.

Why is distributed tracing important?edit

Distributed tracing enables you to analyze performance throughout your microservice architecture by tracing the entirety of a request — from the initial web request on your front-end service all the way to database queries made on your back-end services.

Tracking requests as they propagate through your services provides an end-to-end picture of where your application is spending time, where errors are occurring, and where bottlenecks are forming. Distributed tracing eliminates individual service’s data silos and reveals what’s happening outside of service borders.

For supported technologies, distributed tracing works out-of-the-box, with no additional configuration required.

How distributed tracing worksedit

Distributed tracing works by injecting a custom traceparent HTTP header into outgoing requests. This header includes information, like trace-id, which is used to identify the current trace, and parent-id, which is used to identify the parent of the current span on incoming requests or the current span on an outgoing request.

When a service is working on a request, it checks for the existence of this HTTP header. If it’s missing, the service starts a new trace. If it exists, the service ensures the current action is added as a child of the existing trace, and continues to propagate the trace.

Trace propagation examplesedit

In this example, Elastic’s Ruby agent communicates with Elastic’s Java agent. Both support the traceparent header, and trace data is successfully propagated.

In this example, Elastic’s Ruby agent communicates with OpenTelemetry’s Java agent. Both support the traceparent header, and trace data is successfully propagated.

In this example, the trace meets a piece of middleware that doesn’t propagate the traceparent header. The distributed trace ends and any further communication will result in a new trace.

W3C Trace Context specificationedit

All Elastic agents now support the official W3C Trace Context specification and traceparent header. See the table below for the minimum required agent version:

Agent name	Agent Version
Go Agent	≥`1.6`
Java Agent	≥`1.14`
.NET Agent	≥`1.3`
Node.js Agent	≥`3.4`
PHP Agent	≥`1.0`
Python Agent	≥`5.4`
Ruby Agent	≥`3.5`

Older Elastic agents use a unique elastic-apm-traceparent header. For backward-compatibility purposes, new versions of Elastic agents still support this header.

Visualize distributed tracingedit

APM’s timeline visualization provides a visual deep-dive into each of your application’s traces:

Example view of the distributed tracing in Elastic APM

Manual distributed tracingedit

Elastic agents automatically propagate distributed tracing context for supported technologies. If your service communicates over a different, unsupported protocol, you can manually propagate distributed tracing context from a sending service to a receiving service with each agent’s API.

Add the traceparent header to outgoing requestsedit

Sending services must add the traceparent header to outgoing requests.

Start a transaction with StartTransaction or a span with StartSpan.
Get the active TraceContext.
Send the TraceContext to the receiving service.

Example:

transaction := apm.DefaultTracer.StartTransaction("GET /", "request")   
traceContext := transaction.TraceContext()   

// Send TraceContext to receiving service
traceparent := apmhttp.FormatTraceparentHeader(traceContext))   
tracestate := traceContext.State.String()

	Start a transaction
	Get `TraceContext` from current Transaction
	Format the `TraceContext` or `tracestate` as a `traceparent` header.

Start a transaction with startTransaction, or a span with startSpan.
Inject the traceparent header into the request object with injectTraceHeaders

Example of manually instrumenting an RPC framework:

// Hook into a callback provided by the RPC framework that is called on outgoing requests
public Response onOutgoingRequest(Request request) throws Exception {
  Span span = ElasticApm.currentSpan()   
          .startSpan("external", "http", null)
          .setName(request.getMethod() + " " + request.getHost());
  try (final Scope scope = transaction.activate()) {
      span.injectTraceHeaders((name, value) -> request.addHeader(name, value));   
      return request.execute();
  } catch (Exception e) {
      span.captureException(e);
      throw e;
  } finally {
      span.end();   
  }
}

	Create a span representing an external call
	Inject the `traceparent` header into the request object
	End the span

Start a transaction with apm.startTransaction(), or a span with apm.startSpan().
Get the serialized traceparent string of the started transaction/span with currentTraceparent.
Encode the traceparent and send it to the receiving service inside your regular request.

Example using raw UDP to communicate between two services, A and B:

agent.startTransaction('my-service-a-transaction');   
const traceparent = agent.currentTraceparent;   
sendMetadata(`traceparent: ${traceparent}\n`);

	Start a transaction
	Get the current `traceparent`
	Send the `traceparent` as a header to service B.

Parse the traceparent header on incoming requestsedit

Receiving services must parse the incoming traceparent header, and start a new transaction or span as a child of the received context.

Parse the incoming TraceContext with ParseTraceparentHeader or ParseTracestateHeader.
Start a new transaction or span as a child of the incoming transaction with StartTransactionOptions or StartSpanOptions.

Example:

// Receive incoming TraceContext
traceContext, _ := apmhttp.ParseTraceparentHeader(r.Header.Get("Traceparent"))   
traceContext.State, _ = apmhttp.ParseTracestateHeader(r.Header["Tracestate"]...)   

opts := apm.TransactionOptions{
&#x9;TraceContext: traceContext,   
}
transaction := apm.DefaultTracer.StartTransactionOptions("GET /", "request", opts)

	Parse the `TraceParent` header
	Parse the `Tracestate` header
	Set the parent trace context
	Start a new transaction as a child of the received `TraceContext`

Create a transaction as a child of the incoming transaction with startTransactionWithRemoteParent().
Start and name the transaction with activate() and setName().

Example:

// Hook into a callback provided by the framework that is called on incoming requests
public Response onIncomingRequest(Request request) throws Exception {
    // creates a transaction representing the server-side handling of the request
    Transaction transaction = ElasticApm.startTransactionWithRemoteParent(request::getHeader, request::getHeaders);   
    try (final Scope scope = transaction.activate()) {   
        String name = "a useful name like ClassName#methodName where the request is handled";
        transaction.setName(name);   
        transaction.setType(Transaction.TYPE_REQUEST);   
        return request.handle();
    } catch (Exception e) {
        transaction.captureException(e);
        throw e;
    } finally {
        transaction.end();   
    }
}

	Create a transaction as the child of a remote parent
	Activate the transaction
	Name the transaction
	Add a transaction type
	Eventually, end the transaction

Decode and store the traceparent in the receiving service.
Pass in the traceparent as the childOf option to manually start a new transaction as a child of the received traceparent with apm.startTransaction().

Example receiving a traceparent over raw UDP:

const traceparent = readTraceparentFromUDPPacket()   
agent.startTransaction('my-service-b-transaction', { childOf: traceparent })

	Read the `traceparent` from the incoming request.
	Use the `traceparent` to initialize a new transaction that is a child of the original `traceparent`.

Receive the distributed tracing data on the server side.
Begin a new transaction using the agent’s public API. For example, use ElasticApm::beginCurrentTransaction and pass the received distributed tracing data (serialized as string) as a parameter. This will create a new transaction as a child of the incoming trace context.
Don’t forget to eventually end the transaction on the server side.

Example:

$receiverTransaction = ElasticApm::beginCurrentTransaction(   
    'GET /data-api',
    'data-layer',
    /* timestamp */ null,
    $distDataAsString   
);

	Start a new transaction
	Pass in the received distributed tracing data (serialized as string)

Once this new transaction has been created in the receiving service, you can create child spans, or use any other agent API methods as you typically would.

Create a TraceParent object from a string or HTTP header.
Start a new transaction as a child of the TraceParent by passing in a TraceParent object.

Example using HTTP headers:

parent = elasticapm.trace_parent_from_headers(headers_dict)   
client.begin_transaction('processors', trace_parent=parent)

	Create a `TraceParent` object from HTTP headers formed as a dictionary
	Begin a new transaction as a child of the received `TraceParent`

See the TraceParent API for additional examples.

Start a new transaction or span as a child of the incoming transaction or span with with_transaction or with_span.

Example:

# env being a Rack env
context = ElasticAPM::TraceContext.parse(env: env)   

ElasticAPM.with_transaction("Do things", trace_context: context) do   
  ElasticAPM.with_span("Do nested thing", trace_context: context) do   
  end
end

	Parse the incoming `TraceContext`
	Create a transaction as a child of the incoming `TraceContext`
	Create a span as a child of the newly created transaction. `trace_context` is optional here, as spans are automatically created as a child of their parent’s transaction’s `TraceContext` when none is passed.

« Span compression Transaction sampling »

	Start a new transaction
	Return the string representation of the current transaction’s `TraceParent` object