To shade or not to shade

By
David Pilato

Before version 2.0, Elasticsearch was provided as a JAR with some (but not all) common dependencies shaded and packaged within the same artifact. This helped Java users who embed Elasticsearch in their own applications to avoid version conflicts of modules like Guava, Joda, Jackson, etc. Of course, there was still a list of other unshaded dependencies like Lucene that could still cause conflicts.

Unfortunately, shading is a complex and error prone process which solved problems for some people while creating problems for others. Shading makes it very difficult for developers and plugin authors to write and debug code properly because packages are renamed during the build. Finally, we used to test Elasticsearch unshaded then ship the shaded jar, and we don’t like to ship anything that we aren’t testing.

We have decided to ship Elasticsearch without shading from 2.0 onwards.

Dealing with version conflicts

If you want to use Elasticsearch in your Java application, you may have to deal with version conflicts with third party dependencies like Guava and Joda.  For instance, perhaps Elasticsearch uses Joda 2.8, while your code uses Joda 2.1.  You have two choices:

The simplest solution is to upgrade. Newer module versions are likely to have fixed old bugs. The further behind you fall, the harder it will be to upgrade later. Of course, it is possible that you are using a third party dependency that in turn depends on an outdated version of a package, which prevents you from upgrading.  

The second option is to relocate the troublesome dependencies and to shade them either with your own application or with Elasticsearch and any plugins needed by the Elasticsearch client.

How to shade Elasticsearch

To help you achieve this, we have put together an example application which allows you to run:

  • the Elasticsearch transport client
  • with the Shield security plugin
  • and a conflicting version of Joda.

Your project’s pom.xml probably looks something like this: 

<properties>
   <elasticsearch.version>2.0.0-beta2</elasticsearch.version>
</properties>
<dependencies>
 <dependency>
     <groupId>org.elasticsearch</groupId>
     <artifactId>elasticsearch</artifactId>
     <version>${elasticsearch.version}</version>
 </dependency>
 <dependency>
     <groupId>org.elasticsearch.plugin</groupId>
     <artifactId>shield</artifactId>
     <version>${elasticsearch.version}</version>
 </dependency>
 <dependency>
     <groupId>joda-time</groupId>
     <artifactId>joda-time</artifactId>
     <version>2.1</version>
 </dependency>
</dependencies>

The problem with the above is the Joda dependency: your project requires Joda 2.1, but Elasticsearch 2.0.0-beta2 requires Joda 2.8.

Shading Elasticsearch

To resolve this situation, we create a new maven project that shades Elasticsearch and Shield.  The pom.xml should look like this: 

<groupId>my.elasticsearch.test</groupId>
<artifactId>es-shaded</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
   <elasticsearch.version>2.0.0-beta2</elasticsearch.version>
</properties>
<dependencies>
   <dependency>
       <groupId>org.elasticsearch</groupId>
       <artifactId>elasticsearch</artifactId>
       <version>${elasticsearch.version}</version>
   </dependency>
   <dependency>
       <groupId>org.elasticsearch.plugin</groupId>
       <artifactId>shield</artifactId>
       <version>${elasticsearch.version}</version>
   </dependency>
</dependencies>
<repositories>
   <repository>
       <id>elasticsearch-releases</id>
       http://maven.elasticsearch.org/releases
       <releases>
           <enabled>true</enabled>
           <updatePolicy>daily</updatePolicy>
       </releases>
       <snapshots>
           <enabled>false</enabled>
       </snapshots>
   </repository>
</repositories>

Shade and relocate conflicting packages

Now shade and relocate all the packages which conflict with your own application by adding something like the below to the pom.xml.  This example adds Joda 2.8: 

<build>
   <plugins>
     <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-shade-plugin</artifactId>
       <version>2.4.1</version>
       <executions>
         <execution>
           <phase>package</phase>
           <goals>
             <goal>shade</goal>
           </goals>
           <configuration>
             <relocations>
               <relocation>
                 <pattern>org.joda</pattern>
                 <shadedPattern>my.elasticsearch.joda</shadedPattern>
               </relocation>
             </relocations>
             <transformers>
               <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />
             </transformers>
           </configuration>
         </execution>
       </executions>
     </plugin>
   </plugins>
</build>

Running mvn clean install will create a shaded version of Elasticsearch, Shield, and Joda 2.8 which you can depend on in your application.

Embed this jar within your application

In your project, you can now depend on: 

<!-- 
 This artifact contains all elasticsearch libs including 
 the shaded version of Joda and Shield 
-->
<dependency>
   <groupId>my.elasticsearch.test</groupId>
   <artifactId>es-shaded</artifactId>
   <version>1.0-SNAPSHOT</version>
</dependency>
<!-- 
 You still use your own Joda version 
-->
<dependency>
   <groupId>joda-time</groupId>
   <artifactId>joda-time</artifactId>
   <version>2.1</version>
</dependency>

You can build and use the Elasticsearch TransportClient as before: 

TransportClient client = TransportClient.builder()
       .settings(Settings.builder()
             .put("path.home", ".")
             .put("shield.user", "username:password")
             .put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
       )
       .build();
client.addTransportAddress(new InetSocketTransportAddress(new InetSocketAddress("localhost", 9300)));

To use your own version of Joda, just import org.joda.time.DateTime. You can even access the shaded version of Joda by importing my.elasticsearch.joda.time.DateTime, although we don’t recommend doing so. Here’s an example showing how both versions can be accessed within the same JVM: 

// import org.joda.time.DateTime;
// import my.elasticsearch.joda.time.DateTime;
CodeSource codeSource = new org.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("unshaded = " + codeSource);
codeSource = new my.elasticsearch.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("shaded = " + codeSource);

It will print: 

unshaded = (file:/path/to/joda-time-2.1.jar <no signer certificates>)
shaded = (file:/path/to/es-shaded-1.0-SNAPSHOT.jar <no signer certificates>)

Take a look at this demo project which shows a full running example of the above.

Future developments

Dealing with version conflicts is a perennial problem, but in the future we would like to make it easier than it is today. The simplest way to reduce conflicts between third party dependencies is to reduce the number of dependencies that Elasticsearch has.  We will never be able to remove all dependencies, but we’re making a start by removing Guava in favour of native Java 8 code. We will also be investigating replacing Joda with Java.time (also in Java 8), and possibly even creating a thin Java client for Elasticsearch with minimal dependencies.


Of course, these changes are not small.  Whatever we end up deciding is the right path to follow in the long term, it will take time for us to reach the destination.  In the meantime, we wanted to provide a relatively simple solution for today’s problems.