Regression testing your Java Agent Plugin

For developers who’ve created their own instrumentation with the Java Agent plugin, the next phase of the process is regression testing. By performing regression testing, you can ensure that your plugin functions the way it’s supposed to after you’ve made code changes or updates. 

You’ll have your own plugin, but to illustrate regression testing in this article, I’ll use the plugin in our example repo.

The normal sequence of activity with a plugin

First, let’s understand the normal sequence of activity when the application is started with the Elastic APM (application performance monitoring) Java Agent and the plugin, using the -javaagent option on the command line (There are several ways to attach the agent. For this sequence, let’s assume the recommended -javaagent option.)

1. The Java Virtual Machine (JVM) starts

2. The agent is loaded

3. The agent loads the plugin

4. The application classes are loaded

5. The agent instruments the application

6. The application runs some code that exercises the instrumented methods

7. The agent captures the information that the instrumentation provides

8. The agent communicates with the APM server, sending the information captured from the instrumented method

9. You can view the traces and the information in Elastic APM

We want to validate that the information collected by the agent is indeed what is expected by the test. What we need for testing is some mechanism to provide the information that the agent captures, but that’s also accessible back in the test rather than in Elastic APM. 

There are many ways to approach that — for example, we could mock the agent and send the information back to the test. But knowing that the agent-to-APM-server communication is simple JSON over HTTP, an easier option is to mock the APM server. With a mock APM server, it’s straightforward to capture information sent from the agent and check that it holds the expected results from the instrumentation and test run.

[Related article: Monitoring Elastic Enterprise Search performance using Elastic APM]

The test sequence of activity

With our mock APM server, the sequence of activity for running tests is very similar to the previous sequence, but the last steps change from manual inspection of the information in  Elastic APM to automated inspection by the test.

1. The JVM starts

2. The agent is loaded

3. The agent loads the plugin

4. The application test classes are loaded

5. The agent instruments the application

6. The application runs some test code that exercises the instrumented methods

7. The agent captures the information that the instrumentation provides

8. The agent communicates with a mock APM server, sending the information captured by the instrumented method

9. The test accesses the information from the mock APM server

10. The test validates that the information is what was expected

Implementing the test framework

Let’s break down those steps in more detail to decide how to create a test framework that runs a regression test.

1. The JVM starts. This is initiated by the usual test framework (let’s assume JUnit), so nothing specific to do here.

2. The agent is loaded. We would like to specify the -javaagent option, but at this stage the JVM has already started, so it’s too late. But not to worry — the Elastic APM Java Agent has alternative agent attaching options, and the ElasticApmAttacher.attach() mechanism is perfect here. There is a slight gotcha: we need to specify the APM server URL to the agent, so we need to start the mock APM server first. Consequently, our steps here are:

1. Start the mock APM server

2. Get the (valid system assigned) port it started on

3. Set the server_url property to localhost on that port

4. Start the agent using ElasticApmAttacher.attach()

5. (This is exactly the sequence implemented in AbstractInstrumentationTest.startApmServerAndSetAgentPropertiesAndStartAgent())

3. The agent loads the plugin. Because this is a regression test in the project, we can use the jar generated by the build, which is in the “target” directory of the project. We are building the full jar with all dependencies in the project, so the jar is usable as a plugin (it doesn’t matter what name the jar has, so build version changes are immaterial). However:

1. Maven would run tests in the following sequence with a normal build: 

1. Build the plain jar (without dependencies)

2. Run unit tests

3. Build the target plugin jar (jar-with-dependencies)

4. Run integration tests

2. So working with maven, our actual steps here are:

1. No need to build the plain jar

2. Don’t use non-integration tests

3. Tell the agent to look in the “target” directory for the plugin jar (this is done using the plugins_dir configuration option in AbstractInstrumentationTest.startApmServerAndSetAgentPropertiesAndStartAgent())

4. Make sure we only run the plugin regression tests after the jar-with-dependencies jar is generated, ie in integration test classes, classes ending in IT (“*IT.java” files)

4. The application test classes are loaded. This happens as normal when the test is executed.

5. The agent instruments the application. This should happen as normal when the test is executed, but we’ll check for this when we validate the information sent by the agent.

6. The application runs some test code that exercises the instrumented methods. This should happen as normal when the test is executed, but we’ll check for this when we validate the information sent by the agent.

7. The agent captures the information that the instrumentation provides. This should happen as normal when the test is executed, but we’ll check for this when we validate the information sent by the agent.

8. The agent communicates with a mock APM server, sending the information captured by the instrumented method. This should happen and we’ll test for it, but there’s a race condition here. The agent communicates to the APM server asynchronously (so as not to delay the application at all). Even though it’s a local loopback connection, there is still a slight delay and also a potential batching delay. So we need to do a couple of things here:

1. Minimize any agent-APM server communication batching delay by setting the api_request_size to a small value (this is done in AbstractInstrumentationTest.startApmServerAndSetAgentPropertiesAndStartAgent())

2. Be prepared to wait a bit to retrieve the information from the mock APM server (by adding a timeout to retrieving the information, see the MockApmServer.getAndRemoveTransaction() method)

9. The test accesses the information from the mock APM server.

10. The test validates that the information is what was expected. The mock APM server returns the information in JSON format, so any information can be easily navigated to check it.

Putting it all together

The regression test in the repo holding the example holds three classes that between them implement the test framework and the integration tests. They are small classes and straightforward to understand:

• The MockApmServer is just a webserver that will try to convert incoming text lines into JSON objects. If it finds a JSON object with a root node named “transaction,” it will store the node in an array that can be accessed by anything that holds a reference to the mock APM server object (i.e., subclasses of AbstractInstrumentationTest).
• AbstractInstrumentationTest provides the basis for an integration test, starting the mock APM server and attaching the elastic agent, using all the correct properties needed (and shutting down correctly when the integration test is complete).
• ExampleHttpServerInstrumentationIT is the concrete integration test that runs the full test suite for the plugin by:
  • Starting the application
  • Executing some (HTTP) requests against the application server
  • Checking that the test requests generage the correct number of valid Elastic APM transactions with the values that correspond to those that the instrumentation is setting.
• A pom puts it all together, building and running the test suite on a build (e.g., with mvn clean install).

You can also run the tests from an integrated development environment (IDE) as normal, though the IDE may not build the jar-with-dependencies automatically, so that may need to be configured or built externally.

Try it out

Simply clone the example repo, cd to the root directory of the project, and run mvn clean install. If your test validates that the information was what was expected, then you’re good to go!

The best place to get started with Elastic APM is in the cloud. Begin your 14-day free trial of Elastic Cloud today.