Appendix C: Breaking Changes

Support for the following less popular technologies has been removed in elasticsearch-hadoop 8.9.

We’ve increased the es.scroll.size default from 50 to 1000. If you deploy ES-hadoop in a low memory environment, consider lowering es.scroll.size to avoid issues.

Spark 1.x is no longer maintained. Spark 2 and Spark 3 are still supported.

Spark deprecated support for Scala 2.10 in the Spark 2.0.0 release and removed it in the 2.3.0 release. Scala 2.11 and 2.12 are supported for Spark 2.x.

Hadoop 1.x is no longer maintained. Support for it has not been tested since elasticsearch-hadoop 6.0. Support is now formally deprecated. Hadoop 2 and Hadoop 3 are supported.

Apache Pig is no longer maintained.

Apache Storm has not been a popular Elasticsearch integration.

Each integration that we support incurs a cost for testing and maintenance as new features are added in ES-Hadoop. We’ve seen the usage statistics dropping for the Cascading integration over the last couple of years. These include download numbers, community posts, and open issues. Additionally, ES-Hadoop only supports the Cascading 2 line. Cascading is already into it’s 3.X line and we haven’t seen much interest or pressure to get on to any of the newer versions. Due to these factors, Cascading was deprecated in the 6.6 release, and has been removed in the 7.0.0 release.

ES-Hadoop 7.0.0 and above requires a minimum Java version of 1.8 or above. The library will no longer be compiled for Java 1.6 or Java 1.7.

es.input.use.sliced.partitions has been removed.

es.input.use.sliced.partitions is deprecated in 6.5.0, and will be removed in 7.0.0. The default value for es.input.max.docs.per.partition (100000) will also be removed in 7.0.0, thus disabling sliced scrolls by default, and switching them to be an explicitly opt-in feature.

es.update.script has been deprecated in favor of the new es.update.script. inline option. The es.update.script configuration will continue to function as a legacy configuration with the same meaning as es.update.script.inline.

Coming soon in 6.0 is the removal of multiple types per index. All project tests were migrated to using single types where possible. This migration unearthed a problem with Pig and Cascading in regards to how those frameworks interact with the provided index/type names. During setup, Pig and Cascading attempt to parse these index names as paths, and when the index has a formatted pattern in it (like index-{event:yyyy.MM.dd}/type) the path parsing breaks. Going forward, the colon character : that delimits the field name from the format will be changed to the pipe character |. An example of the new patterned index names is index-{event|yyyy.MM.dd}/type. 5.x will support both separators, but will log a deprecation message when a colon : is encountered.

Elasticsearch 6.0 is seeing the removal of the TTL and Timestamp meta-fields from being allowed on document index and update requests. Because of this, we will be removing the ability to specify them in 6.0 and deprecating their usage in 5.x.

As of 6.0.0 we will no longer be testing on Hadoop 1.x versions. These versions have been considered end of life for quite a while. While the connector may continue to work for those versions, we no longer officially support them. Use these versions at your own risk!

Due to the limitations in YARN for support of long running processes, we have decided to cease development and discontinue the YARN integration in ES-Hadoop. We thank the community for participating in the beta. If you have any questions or concerns about this, please feel free to discuss them on our forums.

In 6.0.0, our ES-Hadoop jar will ship with Spark compiled against Scala 2.11 by default. We will continue to support Scala 2.10 in compatibility artifacts, but the main project artifact will target Scala 2.11 by default.

ES-Hadoop 5.0 has added support for new versions of software from the Hadoop ecosystem. Support for older versions is normally dropped during these cycles for the purposes of stability and to allow for cleanup of older code. Please note the following version compatibility changes in 5.0:

With the removal of support for Spark 1.0-1.2 and the addition of support for Spark 2.0, the names of all the Spark artifacts included in ES-Hadoop 5.0 have changed. All Spark artifacts in this release now have both of their Spark and Scala versions explicitly demarcated in their names (instead of just Scala and sometimes Spark).

Table1: Spark Jar Name Changes from 2.x to 5.x

In 5.0, the HDFS Repository plugin has been all but re-built from the ground up to allow for compatibility with the changes to security policies in Elasticsearch’s plugin framework.