5.4.0 Release Notes

edit

See Multi data path bug in Elasticsearch 5.3.0.

Also see Breaking changes in 5.4.

Breaking changes

edit
Settings

Breaking Java changes

edit
Aggregations
  • Move getProperty method out of MultiBucketsAggregation.Bucket interface #23988
  • Remove getProperty method from Aggregations interface and impl #23972
  • Move getProperty method out of Aggregation interface #23949
Java API
  • Fold InternalSearchHits and friends into their interfaces #23042

Deprecations

edit
Aggregations
  • Deprecate Stats#getCountAsString #24292
Java API
  • Add BulkProcessor methods with XContentType parameter #23078 (issue: #22691)
Network
Packaging
  • Add deprecation warnings for $ES_USER and $ES_GROUP #24190 (issue: #23989)
Plugin Delete By Query
  • Deprecate delete_by_query requests without an explicit query #23635
Plugin Repository Azure
  • Deprecate global repositories.azure settings #22856 (issue: #22800)
Plugin Repository S3
REST
  • Deprecate ldjson support and document ndjson for bulk/msearch #23051 (issues: #23025, #23049)
Stats
  • Deprecate _field_stats endpoint #23914

New features

edit
Analysis
  • Adds pattern keyword marker filter support #23600 (issue: #4877)
  • Expose WordDelimiterGraphTokenFilter #23327 (issue: #23104)
Index APIs
  • Add FieldCapabilities (_field_caps) API #23007 (issue: #22438)
Search
  • Introduce incremental reduction of TopDocs #23946
Similarities
  • Adds boolean similarity to Elasticsearch #23637 (issue: #6731)

Enhancements

edit
Aggregations
  • Add BucketMetricValue interface #24188
  • Move aggs CommonFields and TYPED_KEYS_DELIMITER from InternalAggregation to Aggregation #23987
  • Use ParseField for aggs CommonFields rather than String #23717
  • Share XContent rendering code in terms aggs #23680
  • Add unit tests for ParentToChildAggregator #23305 (issue: #22278)
  • First step towards incremental reduction of query responses #23253
Allocation
  • Trigger replica recovery restarts by master when primary relocation completes #23926 (issue: #23904)
  • Makes the same_shard host dynamically updatable #23397 (issue: #22992)
Analysis
  • Support Keyword type in Analyze API #23161
Cluster
  • Prevent nodes from joining if newer indices exist in the cluster #23843
Core
  • Detect remnants of path.data/default.path.data bug #24099 (issues: #23981, #24052, #24074, #24093)
  • Await termination after shutting down executors #23889
  • Add early-access check #23743 (issue: #23668)
  • Adapter action future should restore interrupts #23618 (issue: #23617)
  • Disable bootstrap checks for single-node discovery #23598 (issues: #23585, #23595)
  • Enable explicitly enforcing bootstrap checks #23585 (issue: #21864)
  • Add equals/hashcode method to ReplicationResponse #23215
Dates
  • Improve error handling for epoch format parser with time zone (#22621) #23689
Discovery
  • Introduce single-node discovery #23595
  • UnicastZenPing shouldn’t ping the address of the local node #23567
  • MasterFaultDetection can start after the initial cluster state has been processed #23037 (issue: #22828)
Highlighting
  • Add support for fragment_length in the unified highlighter #23431
  • Add BreakIteratorBoundaryScanner support #23248
Index APIs
  • Wildcard cluster names for cross cluster search #23985 (issue: #23893)
Ingest
  • Lazy load the geoip databases #23337
Internal
  • Add a dedicated TransportRemoteInfoAction for consistency #24040 (issue: #23969)
  • Simplify sorted top docs merging in SearchPhaseController #23881
  • Synchronized CollapseTopFieldDocs with lucenes relatives #23854
  • Cleanup SearchPhaseController interface #23844
  • Do not create String instances in Strings methods accepting StringBuilder #22907
Java API
  • Added types options to DeleteByQueryRequest #23265 (issue: #21984)
Java High Level REST Client
  • Convert suggestion response parsing to use NamedXContentRegistry #23355
  • UpdateRequest implements ToXContent #23289
  • Add javadoc for DocWriteResponse.Builders #23267
  • Expose WriteRequest.RefreshPolicy string representation #23106
  • Use typed_keys parameter to prefix suggester names by type in search responses #23080 (issue: #22965)
  • Add parsing methods to BulkItemResponse #22859
Logging
  • Warn on not enough masters during election #20063 (issue: #8362)
Mapping
  • Improve error message for ipv6 on legacy ip fields #23136 (issue: #23126)
Nested Docs
  • Avoid adding unnecessary nested filters when ranges are used. #23427
Network
  • Adjust default Netty receive predictor size to 64k #23542 (issue: #23185)
  • Keep the pipeline handler queue small initially #23335
  • Set network receive predictor size to 32kb #23284 (issue: #23185)
  • TransportService.connectToNode should validate remote node ID #22828 (issue: #22194)
Packaging
  • Introduce Java version check #23194 (issue: #21102)
  • Cleanup some things after removal of joda-time hack #18959
Percolator
  • Allowing range queries with now ranges inside percolator queries #23921 (issue: #23859)
  • Add term extraction support for MultiPhraseQuery #23176
Plugin Discovery EC2
  • Settings: Migrate ec2 discovery sensitive settings to elasticsearch keystore #23961 (issue: #22475)
Plugin Lang Painless
  • Allow painless to load stored fields #24290
  • Start on custom whitelists for Painless #23563
  • Fix Painless’s implementation of interfaces returning primitives #23298 (issue: #22983)
  • Allow painless to implement more interfaces #22983
Plugin Repository Azure
  • Add Backoff policy to azure repository #23387 (issue: #22728)
Plugin Repository S3
  • Removes the retry mechanism from the S3 blob store #23952 (issue: #22845)
  • S3 Repository: Eagerly load static settings #23910
Plugins
  • Modify permissions dialog for plugins #23742
  • Plugins: Add plugin cli specific exit codes #23599 (issue: #15295)
  • Plugins: Output better error message when existing plugin is incompatible #23562 (issue: #20691)
Query DSL
  • Make it possible to validate a query on all shards instead of a single random shard #23697 (issue: #18254)
REST
  • Validate top-level keys when parsing mget requests #23746 (issue: #23720)
  • Cluster stats should not render empty http/transport types #23735
  • Add parameter to prefix aggs name with type in search responses #22965
Search
  • Set shard count limit to unlimited #24012
  • Streamline shard index availability in all SearchPhaseResults #23788
  • Search took time should use a relative clock #23662
  • Prevent negative from parameter in SearchSourceBuilder #23358 (issue: #23324)
  • Remove unnecessary result sorting in SearchPhaseController #23321
  • Expose batched_reduce_size via _search #23288 (issue: #23253)
  • Adding fromXContent to Suggest and Suggestion class #23226 (issue: #23202)
  • Adding fromXContent to Suggestion.Entry and subclasses #23202
  • Add CollapseSearchPhase as a successor for the FetchSearchPhase #23165
  • Integrate IndexOrDocValuesQuery. #23119
  • Detach SearchPhases from AbstractSearchAsyncAction #23118
  • Fix GraphQuery expectation after Lucene upgrade to 6.5 #23117 (issue: #23102)
  • Nested queries should avoid adding unnecessary filters when possible. #23079 (issue: #20797)
  • Add xcontent parsing to completion suggestion option #23071
  • Add xcontent parsing to suggestion options #23018
  • Separate reduce (aggs, suggest and profile) from merging fetched hits #23017
Settings
  • Add secure file setting to keystore #24001
  • Add a setting which specifies a list of setting #23883
  • Add a property to mark setting as final #23872
  • Remove obsolete index setting index.version.minimum_compatible. #23593
  • Provide a method to retrieve a closeable char[] from a SecureString #23389
  • Update indices settings api to support CBOR and SMILE format #23309 (issues: #23242, #23245)
  • Improve setting deprecation message #23156 (issue: #22849)
Snapshot/Restore
  • Change snapshot status error to use generic SnapshotException #24355 (issue: #24225)
Stats
  • Add cross-cluster search remote cluster info API #23969 (issue: #23925)
Task Manager
  • Allow task to be unregistered by ClusterStateApplier #23931
  • Limit IndexRequest toString() length #22832

Bug fixes

edit
Aggregations
  • Align behavior HDR percentiles iterator with percentile() method #24206
  • The filter and significant_terms aggregations should parse the filter as a filter, not a query. #23797
  • Completion suggestion should also consider text if prefix/regex is missing #23451 (issue: #23340)
  • Fixes the per term error in the terms aggregation #23399
  • Fixes terms error count for multiple reduce phases #23291 (issue: #23286)
  • Restore support for the include/pattern syntax. #23140 (issue: #22933)
Bulk
CRUD
Cluster
  • Don’t set local node on cluster state used for node join validation #23311 (issues: #21830, #3, #4, #6, #9)
  • Allow a cluster state applier to create an observer and wait for a better state #23132 (issue: #21817)
  • Cluster allocation explain to never return empty response body #23054
Core
  • Check for default.path.data included in path.data #24285 (issue: #24283)
  • Improve performance of extracting warning value #24114 (issue: #24018)
  • Reject duplicate settings on the command line #24053
  • Restrict build info loading to ES jar, not any jar #24049 (issue: #21955)
Discovery
  • ZenDiscovery - only validate min_master_nodes values if local node is master #23915 (issue: #23695)
Index APIs
  • Fixes restore of a shrunken index when initial recovery node is gone #24322 (issue: #24257)
  • Honor update request timeout #23825
Ingest
  • Improve missing ingest processor error #23379 (issue: #23392)
  • update _ingest.timestamp to use new ZonedDateTime #23174 (issue: #23168)
Inner Hits
  • Replace NestedChildrenQuery with ParentChildrenBlockJoinQuery #24016 (issue: #24009)
  • Changed DisMaxQueryBuilder to extract inner hits from leaf queries #23512 (issue: #23482)
Internal
  • Add infrastructure to mark contexts as system contexts #23830
  • Always restore the ThreadContext for operations delayed due to a block #23349
Java High Level REST Client
  • Correctly parse BulkItemResponse.Failure’s status #23432
Java REST Client
  • Make buffer limit configurable in HeapBufferedConsumerFactory #23970 (issue: #23958)
  • RestClient asynchronous execution should not throw exceptions #23307
Mapping
  • Preserve response headers when creating an index #23950 (issue: #23947)
  • Improves disabled fielddata error message #23841 (issue: #22768)
  • Switch include_in_all in multifield to warning #23656 (issues: #21971, #23654)
  • Fix MapperService StackOverflowError #23605 (issue: #23604)
  • Fix NPE with scaled floats stats when field is not indexed #23528 (issue: #23487)
Network
  • Fix possible hang in local transport when nodes get concurrently disconnected #23962 (issue: #23942)
  • Respect promises on pipelined responses #23317 (issues: #23310, #23322)
  • Ensure that releasing listener is called #23310
Packaging
  • Fall back to non-atomic move when removing plugins #23548 (issue: #35)
Percolator
  • Fix memory leak when percolator uses bitset or field data cache #24115 (issue: #24108)
Plugin Ingest Attachment
Plugin Lang Painless
  • Fix painless’s regex lexer and error messages #23634
  • Replace Painless’s Cast with casting strategies #23369
  • Fix Bad Casts In Painless #23282 (issue: #23238)
Plugin Repository Azure
  • Azure blob store’s readBlob() method first checks if the blob exists #23483 (issue: #23480)
Plugin Repository S3
  • Handle BlobPath’s trailing separator case. Add test cases to BlobPathTests.java #23091
Plugins
  • Fix delete of plugin directory on remove plugin #24266 (issue: #24252)
  • Use a marker file when removing a plugin #24252 (issue: #24231)
Query DSL
  • FuzzyQueryBuilder should error when parsing array of values #23762 (issue: #23759)
REST
  • [API] change wait_for_completion default according to docs #23672
  • Deprecate request_cache for clear-cache #23638 (issue: #22748)
  • HTTP transport stashes the ThreadContext instead of the RestController #23456
  • Ensure we try to autodetect content type for handlers that support plain text #23452
  • Fix date format in warning headers #23418 (issue: #23275)
  • Align REST specs for HEAD requests #23313 (issue: #21125)
  • Correct warning header to be compliant #23275 (issue: #22986)
  • Fix get HEAD requests #23186 (issue: #21125)
  • Fix search scroll request with a plain text body #23183 (issue: #22691)
  • Handle bad HTTP requests #23153 (issue: #23034)
  • Fix get source HEAD requests #23151 (issue: #21125)
  • Properly encode location header #23133 (issues: #21057, #23115)
  • Fix template HEAD requests #23130 (issue: #21125)
  • Fix index HEAD requests #23112 (issue: #21125)
  • Fix alias HEAD requests #23094 (issue: #21125)
Recovery
  • Provide target allocation id as part of start recovery request #24333 (issue: #24167)
Reindex API
  • Fix throttled reindex_from_remote #23953 (issues: #23828, #23945)
  • Fix reindex with a remote source on a version before 2.0.0 #23805
  • Make reindex wait for cleanup before responding #23677 (issue: #23653)
Scripting
  • Remove unnecessary Groovy deprecation logging #23410 (issue: #23401)
  • Convert script/template objects to json format internally #23308 (issue: #23245)
  • Script: Fix value of ctx._now to be current epoch time in milliseconds #23175 (issue: #23169)
Search
  • Cross Cluster Search: propagate original indices per cluster #24328
  • Query string default field #24214
  • Speed up parsing of large terms queries. #24210
  • IndicesQueryCache should delegate the scorerSupplier method. #24209
  • Fork LRUQueryCache from Lucene to work around LUCENE-7749 #24207
  • Disable graph analysis at query time for shingle and cjk filters producing tokens of different size #23920 (issue: #23918)
  • Fix cross-cluster remote node gateway attributes #23863
  • Use a fixed seed for computing term hashCode in TermsSliceQuery #23795
  • Honor max concurrent searches in multi-search #23538 (issue: #23527)
  • Avoid stack overflow in multi-search #23527 (issue: #23523)
  • Fix query_string_query to transform "foo:*" in an exists query on the field name #23433 (issue: #23356)
  • Factor out filling of TopDocs in SearchPhaseController #23380 (issues: #19356, #23357)
  • Replace blocking calls in ExpandCollapseSearchResponseListener by asynchronous requests #23053 (issue: #23048)
Search Templates
  • No longer add illegal content type option to stored search templates #24251 (issue: #24227)
Settings
  • Do not set path.data in environment if not set #24132 (issue: #24099)
  • Correct handling of default and array settings #24074 (issues: #23981, #24052)
  • Fix merge scheduler config settings #23391
  • Settings: Fix keystore cli prompting for yes/no to handle console returning null #23320
Similarities
  • Fix similarity upgrade when "default" similarity is overridden #23163
Snapshot/Restore
  • Fixes maintaining the shards a snapshot is waiting on #24289
  • Fixes snapshot status on failed snapshots #23833 (issue: #23716)
  • Fixes snapshot deletion handling on in-progress snapshot failure #23703 (issue: #23663)
  • Prioritize listing index-N blobs over index.latest in reading snapshots #23333
Stats
  • Avoid overflow when computing total FS stats #23641
  • Handle existence of cgroup version 2 hierarchy #23493 (issue: #23486)
  • Handle long overflow when adding paths' totals #23293 (issue: #23093)
  • Fix control group pattern #23219 (issue: #23218)
  • Fix total disk bytes returning negative value #23093

Regressions

edit
Bulk

Upgrades

edit
Aggregations
Core
  • Upgrade to Lucene 6.5.0 #23750
  • Upgrade from JNA 4.2.2 to JNA 4.4.0 #23636
  • Upgrade to lucene-6.5.0-snapshot-d00c5ca #23385
  • Upgrade to lucene-6.5.0-snapshot-f919485. #23087
Logging
  • Upgrade to Log4j 2.8.2 #23995
Network
Plugin Repository Azure