Elasticsearch version 7.15.0

edit

Elasticsearch version 7.15.0

edit

Also see Breaking changes in 7.15.

Known issues

edit
  • Frozen tier: (Windows only) The frozen data tier relies on multiple caching mechanisms to speed up access to searchable snapshot files. One of these caches uses sparse files to avoid creating large files on disk when it is not strictly required. A bug prevented files from being created with the right options to enable sparse support on Windows, leading Elasticsearch to create potentially large files that can end up consuming all the disk space.

    This issue is fixed in Elasticsearch versions 7.15.2 and later. There is no known workaround for earlier versions. Filesystems that enable sparse files by default are not affected. For more details, see #79371.

  • Aggregations: In Elasticsearch 7.14.0–7.15.0, when a cross-cluster search (CCS) request is proxied, the memory for the aggregations on the proxy node will not be freed. The trigger is cross-cluster search using aggregations where minimize roundtrips is not effective (for example, when minimize roundtrips is explicitly disabled, or implicitly disabled when using scroll, async and point-in-time searches).

    This affects Kibana CCS aggregations because Kibana uses async search by default. This issue can also happen in all modes of remote connections configured for cross-cluster search (sniff and proxy). In sniff mode, we only connect to a subset of the remote nodes (by default 3). So if the remote node we want to send a request to is not one of those 3, we must send the request as a proxy request. The workaround is to periodically restart nodes with heap pressure.

    We have fixed this issue in Elasticsearch 7.15.1 and later versions. For more details, see #78404.

  • Parsing a request when the last element in an array is filtered out (for instance using _source_includes) fails. This is due to a bug in Jackson parser. Fixed in Elasticsearch 8.6.1 (#91456)
  • The deprecated index.mapper.dynamic setting can break your cluster. It can only be set using the Update index settings API. Symptoms include nodes failing to start or shards failing to allocate. Do not use this setting in versions prior to 7.17.22. The bug is fixed in 7.17.22. (issue: #109160)

Breaking changes

edit
Infra/Core
Search

Deprecations

edit
Store
  • Deprecate SimpleFS and replace it with NIOFS #75156

New features

edit
EQL
  • Add multi-valued fields support. Part 1 #74567
  • Backwards compatibility support for multi value fields update #76605
  • Multi-value fields support #76610
Geo
  • Remove vector tile feature flag #76068
Infra/Node Lifecycle
  • Remove Node Shutdown API feature flag #76588 (issue: #70338)
Machine Learning
  • Add model_prune_window field to AD job config #75741 (issue: #1962)
Mapping
  • Unsigned longs should be compatible with index sorting #75599

Enhancements

edit
Allocation
  • Clarify allocation explain if random shard chosen #75670
  • Prefer primaries in cluster allocation explain #76220
Authentication
  • Add kibana-system service account #76449
  • Support 2b and 2y prefixes in bcrypt #76083 (issues: #51132, #56086)
  • Support password_hash for Change Password API #75500
Authorization
  • Add threshold time logging for Authorized indices #75439
  • Fix privileges for GetRollupIndexCapabilities API #75614 (issue: #74779)
  • Granting kibana_system reserved role access to "all" privileges to .internal.alerts* index #76624
Cluster Coordination
  • Distinguish timeouts/failures in follower checker #75519
  • Indicate quorumness of discovered nodes earlier #74823
  • Warn on possible master service starvation #74820
Data streams
  • Add filter support to data stream aliases #74784 (issue: #66163)
EQL
  • Minimise CCS roundtrips #76076
Engine
Geo
  • Add mvt field type format to geo fields #75367 (issue: #73872)
ILM+SLM
  • Add phase configs to ILM telemetry #74575 (issue: #69152)
  • Add validation of the number_of_shards parameter in Shrink Action of ILM #74219 (issue: #72724)
  • Allow ILM move-to-step without action or name #75435 (issue: #58128)
  • Update ILM message for out of order phases error #75099 (issue: #70336)
Infra/CLI
  • In MultiCommand, show command list when required #75993
Infra/Logging
  • Adds minimal traceparent header support to Elasticsearch #74210
Infra/Plugins
  • Add support in plugins for a separate spi classloader #76288 (issue: #74448)
Infra/Resiliency
  • Print suggestion to view log on fatal error #75418 (issue: #61767)
Infra/Scripting
  • Add Fields API to aggregation scripts and field scripts #76325
  • Add ability to allow list instance methods on the script class #76045
  • Add fields api to runtime fields contexts #76265
  • Script: Def encoding parser #74840
  • Script: Fields API for Filter context #76119
  • Script: Fields API for Sort and Score scripts #75863 (issue: #61388)
  • Script: Mangle user function names #72892 (issue: #69742)
  • Script: ulong via fields API #76519 (issue: #64361)
Ingest
  • Add support for _meta field to ingest pipelines #75905
  • Configurable media_type for mustache template encoding on append processor #76210
  • Json processor: add_to_root_conflict_strategy option #74967
  • Json processor: allow duplicate keys #74956
  • Map iteration support for ForEach processor #64062 (issue: #55215)
  • Support for wildcards and override option for dot_expander processor #74601
License
  • Add persistent licensed feature tracking #76476
Machine Learning
  • Add datafeed_config field to anomaly detection job configs #74265
  • Adding datafeed_config to job in high level rest client #75338
  • Adding new p_value scoring heuristic to significant terms aggregation #75313
  • Delete expired annotations #75617 (issue: #75572)
  • Deleting a job now deletes the datafeed if necessary #76010
  • Integrating ML with the node shutdown API #75188
  • Throttle job audit msgs if delayed data occurs for consecutive buckets #75815 (issue: #73679)
  • Speed up training of regression and classification models on very large data sets #1941
  • Improve regression and classification training accuracy for small data sets #1960
  • Prune models for split fields (by, partition) that haven’t seen data updates for a given period of time #1962
Mapping
  • Handle runtime subfields when shadowing dynamic mappings #75595 (issue: #75454)
Network
  • Drop ReceiveTimeoutTransportException stack trace #75671
  • Ensure indexing_data is compressed appropriately #76321 (issue: #73497)
  • Respond with same compression scheme received #76372 (issue: #73497)
Other
  • Add resolve_during_rolling_upgrade field to deprecation issue #74226 (issue: #73091)
Packaging
  • Accept settings in snake case in Docker image #74327 (issue: #74036)
  • Change env var prefix in Docker settings support #76192 (issues: #74327, #76148)
  • Reduce default Docker image size in 7.x #75079
  • Tighten up write permissions in Docker image #70635
Query Languages
  • Introduce ParserUtils to consolidate code #76399
Recovery
  • Add peer recoveries using snapshot files when possible #76237 (issue: #73496)
  • Add peer recovery planners that take into account available snapshots #75840
  • Compute latest snapshot directly in TransportGetShardSnapshotAction #76254 (issue: #73496)
  • Keep track of data recovered from snapshots in RecoveryState #76499 (issue: #73496)
Reindex
  • Log reindex bulk request completion #75935
Search
  • Add index disk usage API #74051 (issue: #74051)
  • Allow specifying index in pinned queries #74873 (issue: #67855)
  • Compress async search responses before storing #74766 (issue: #67594)
  • Emit multiple fields from a runtime field script #75108 (issue: #68203)
  • Replaced field_masking_span occurrences with respective ParseField #74718 (issue: #63527)
  • Search performance - better caching logic for queries on wildcard field #76035 (issue: #75848)
  • Support search slicing with point-in-time #74457 (issue: #65740)
  • Track Lucene field usage #74227 (issue: #73944)
  • Unsigned long should accept value with zero fraction #75646 (issue: #67565)
Security
  • A new search API for API keys - core search function #75335 (issues: #71023, #73705)
  • Add sort and pagination support for QueryApiKey API #76144 (issue: #71023)
  • HLRC support for query API key API #76520 (issues: #71023, #75335, #76144)
  • Query API key - Rest spec and yaml tests #76238 (issue: #71023)
  • Return file-backed service tokens from all nodes #75200 (issue: #74530)
  • OnChange listener now only runs when there is actual change #75207 (issue: #44701)
Snapshot/Restore
  • Add the ability to fetch the latest successful shard snapshot #75080 (issue: #73496)
  • Fix SnapshotInfo#toXContent delegation #75899
  • Implement Numeric Offset Parameter in Get Snapshots API #76233 (issue: #74350)
  • Include extra snapshot details in logs/APIs #75917
  • Prevent searchable snapshots indices to be shrunk/split #75227 (issue: #56595)
  • Return Total Result Count and Remaining Count in Get Snapshots Response #76150
Stats
  • Adding shard count to node stats api #75433
  • Aggregate data tier index stats separately from node stats #76322
Store
  • Replace internal usages of SimpleFS with NIOFS #74996
TLS
  • Add KeyUsage, ExtendedKeyUsage, CipherSuite & Protocol to SSL diagnos… #65634 (issue: #63784)
  • Solved issue "certutil: large passwords not set" #30944 #36689
Task Management
  • Adding the option to abort persistent tasks locally #74115
  • Include reason in cancellation exceptions #75332 (issue: #74825)
Transform
  • Align transform checkpoint range with date_histogram interval for better performance #74004 (issue: #62746)
  • Changing hard coded 10k page size limit to 65k #74651 (issue: #57719)
  • Improve performance by using point in time API for search #74984 (issue: #73481)
  • Optimize composite agg execution using ordered groupings #75424

Bug fixes

edit
Aggregations
  • Fix docCountError calculation for multiple reduces #76391 (issues: #43874, #75667)
  • Fix wrong error upper bound when performing incremental reductions #43874 (issue: #40005)
Allocation
  • Active masternode crash caused by datanode Input/output error #76480 (issues: #1, #436)
Distributed
  • Wrap exception with SendRequestTransportException instead of TransportException #73551
Geo
  • Alias field does not work with geo_shape query #74895
  • Use contains relation for geo_point intersection over geo_point field #75272 (issue: #75103)
ILM+SLM
  • Changing wait_for_snapshot to check start time rather than finish time #75644
Indices APIs
  • Fix pending deletes timeout log message #75776
Infra/Core
  • Add system data streams to feature state snapshots #75902 (issue: #75860)
  • Fix for JarHell Bootstrap Check can yield false positives #76217 (issue: #75701)
Infra/Scripting
  • Fix the error message for instance methods that don’t exist #76512
Ingest
  • Execute EnrichPolicyRunner on a non dedicated master node #76881 (issue: #70436)
Machine Learning
  • Allow for larger models in the inference step for data frame analytics #76116 (issue: #76093)
  • Ignore exceptions while opening job after SIGTERM to JVM #75850
  • Prevent accidentally asking for more resources when scaling down and improve scaling size estimations #74691 (issue: #74709)
  • When setting upgrade mode ensure that internal actions don’t throw unnecessary permissions errors #75880
  • Fix potential "process stopped unexpectedly: Fatal error" for training regression and classification models #1997 (issue: #1956)
Network
  • Ensure replica requests are marked as index_data (#75008) #75014 (issue: #73497)
  • Fix memory/breaker leaks for outbound responses (#76474) #76536 (issues: #65921, #76474)
Search
  • Remove invalid path from Open PIT rest spec #77609
  • Skip noop normalizing #74656
Security
  • Don’t remove warning headers on all failure #76434 (issue: #75739)
Snapshot/Restore
  • Fix Get Snapshots Request Cancellation with ignore_unavailable=true #78004 (issue: #77980)
  • Fix Spurious Warnings During Snapshot Delete #75911
  • Include full blob key in Azure read operations exceptions #75920 (issue: #75916)
  • Not throwing FileNotFoundException if BlobContainer has been deleted #75991
Transform
  • Fix potential deadlock when using stop_at_checkpoint #76034 (issue: #75846)

Upgrades

edit
Ingest
  • Upgrade Tika to 1.27 for ingest #75191
Network
  • Upgrade to Netty 4.1.66 #76135
Query Languages
Snapshot/Restore
  • Upgrade GCS SDK to 1.117.1 #74938