WARNING: Version 5.4 has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Elasticsearch for Apache Hadoop version 5.0.1
Elastic Docs ›Elasticsearch for Apache Hadoop [5.4] ›Release Notes

Elasticsearch for Apache Hadoop version 5.0.0

edit

Elasticsearch for Apache Hadoop version 5.0.0

edit

October 26, 2016

Breaking Changes

edit
  • Update to compat with Storm 1.x.x #769
  • Why does EsBolt still use the old JAR #754
  • Drop HDFS repository #741
  • Drop support for Spark 1.0-1.2 #740
  • Remove legacy Hive version #739

New Features

edit
Mapping
  • Add support for 16-bit/half floats in ES 5.x #800
  • New ES type scaled_float #822
MR
  • Integrating with ES ingest node pipeline #806
New Slice API
  • Add the ability to create IndexPartition based on the desired number of documents per split #812 (issue: #778)
Spark
  • Support for Spark 2.0 / Datasets #647
  • Provided dedicated support for Spark 1.3-1.6 DStreams #802

Enhancements

edit
Build
  • 2.3.1 release notes list many bugs as fixed without commits #770
REST
  • Expire Unused Pooled Transports in Spark Streaming #849
  • Validate field exclusion settings when EsSpark.saveJsonToEs() is executed #782
  • Review the error messages in ES 5.x and properly parse them #779
  • Align extended Boolean parsing with Elasticsearch #798
  • Fields replacement in es.update.script.params fails for objects #760
  • Replace/Ignore DOT character in field names before inserting into Elastic Search #758
Spark
  • Improve ElasticsearchRelation extensibility #829
  • Explicitly specify buildScan return type #826
  • Upgrade to Spark 1.6.2 #797
  • Fix example for spark. #768 (issue: #767)

Bug Fixes

edit
Build
  • Repository HDFS still referred in the build system #775
  • Spark 1.2 artifacts are still referenced #774
Hive
  • Date type not properly inserted into ES #757
MR
  • Reading boolean 0/1 values from ES into Spark does not work (although false/true is ok) #795
  • Elasticsearch Mapreduce write "es.mapping.exclude" not working #752
MR and Spark
  • Restore data locality preference for hadoop and spark #819 (issue: #814)
NetworkClient
  • Node [127.0.0.1:9200] failed (Read timed out); no other nodes left - aborting…​ #753
  • Invalid target URI POST@null/pharmadata/test/_search #748
Pig
  • Pig maps can have values of different types #777
Rest
  • Update/Upsert bulk actions with scripts are broken for ES 1.x as of 5.0.0-alpha5 #817
  • Shard size estimations for Slice API do not target shards #843
  • Elasticsearch-spark connector failing to save data with an Illegal Argument Exception : "No class name given" #837
  • Spark Datasource: Non empty BINARY fields are always considered empty #835
  • Fix memory leak caused by re-wrapping DelegatedProtocol #823
  • Fixed parsing of ES mapping with OBJECT field named properties #810 (issue: #809)
  • Not all errors in ES are reported when writing #720
  • es.index.auto.create setting doesn’t work as expected #793
  • Parsing error messages can cause a NPE #776
  • shard preference concatenation with | gives query error #874
Spark
  • Dropping a whole column of a DataFrame while indexing the DF into ES #841
  • Fix: Non empty BINARY fields are considered empty #834
  • saveToEs saves fields with NULL values #792
  • Failed to write data back to ES by using Spark DataFrame save() API !! #749
  • Pushdown option not working as expected with Spark data frames #734
URLDecoder
  • Illegal hex characters in escape (%) pattern - For input string: " S" #747
Yarn
  • Authenticate with a local keytab in the YARN Application Master #807

Documentation

edit
  • es.read.field.as.array.include NOT es.field.read.as.array.include #860
  • Corrected property name #805
  • Repository HDFS doc improvements #781
  • Snapshot Creation Exception #813
  • Corrected link to ADD command #788
  • Update spark.adoc #785
  • Update cloud.adoc #773
  • Update configuration.adoc #772
  • Tests fail in build #755
  • Documentation Issue : Wrong package name #838
  • ES_CLASSPATH removed from ES 2.x #861

Non-issue

edit
Hive
  • Error Loading Data #839
  • EsHadoopIllegalArgumentException: Cannot detect ES version #794
  • ES-Hive throwing exception while reading for bulk no. of columns #790
  • Create a external table with location command success while elasticsearch-hadoop do not support location function #786
Pig
  • Store Map as object with Pig in Elasticsearch #848
  • Get fields with documents in Elasticsearch #825
  • Failure while using EsStorage twice on a single Pig script to store a Parent Child relation #756
  • Store Tuple as object with Pig in Elasticsearch #746
REST
  • Some way to query "total" field from result set? #830
  • PROBLEM: Failed to write data back to ES by using Spark DataFrame save() API !! #836
  • Elasticsearch : Cannot detect ES version #791
Spark
  • Class Not Found Exception #863
  • java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class #862
  • Array types not supported in automatic mapping #847
  • Connection error with Elasticsearch 2.4.0 #846
  • Handling decimal type in dataset #842
  • Fix import of JavaEsSparkSQL #840
  • Got exception when I tried to invoke _mtermvector #796
  • ES-Hadoop on Spark 2.0 #759
  • elasticsearch-hadoop exception about mapping #766
  • Parameter ES_MAPPING_TIMESTAMP is not working using saveToEs #765
« Elasticsearch for Apache Hadoop version 5.0.1