A newer version is available. For the latest information, see the current release documentation.

« Elasticsearch version 7.4.1 Elasticsearch version 7.3.2 »

› ›

Elasticsearch version 7.4.0

edit

Elasticsearch version 7.4.0

edit

Also see Breaking changes in 7.4.

Known issues

edit

Applying deletes or updates on an index after it has been shrunk may corrupt the index. In order to prevent this issue, it is recommended to stop shrinking read-write indices. For read-only indices, it is recommended to force-merge indices after shrinking, which significantly reduces the likeliness of this corruption in the case that deletes/updates would be applied by mistake. This bug is fixed in Elasticsearch 7.7 and later versions. More details can be found on the corresponding issue.
Activating the search slow log should be avoided in this version. Any attempt to log a slow search can throw an AIOOBE due to a bug that performs concurrent modifications on a shared byte array. (issue: #48358)
Indices created in 6.x with date and date_nanos fields using formats that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. https://github.com/elastic/elasticsearch/pull/52555 This is fixed in Elasticsearch 7.7 and later versions.

Breaking changes

edit

Infra/REST API

Update the schema for the REST API specification #42346 (issue: #35262)

Machine Learning

Improve progress reporting for data frame analytics #45856

Ranking

Forbid empty doc values on vector functions #43944

Search

Use float instead of double for query vectors. #46004

Snapshot/Restore

Provide an Option to Use Path-Style-Access with S3 Repo #41966 (issue: #41816)

Transforms

Combine task_state and indexer_state in _stats #45276 (issue: #45201)
Improve response format of transform stats endpoint #44350 (issue: #43767)

Breaking Java changes

edit

Geo

Geo: Change order of parameter in Geometries to lon, lat #45332 (issue: #45048)

Network

Stop Recreating Wrapped Handlers in RestController #44964

Deprecations

edit

Geo

Geo: add Geometry-based query builders to QueryBuilders #45058 (issues: #44715, #45048)

Infra/Core

Bundle AdoptOpenJDK 13 #46860
Add deprecation check for pidfile setting #45939 (issues: #45938, #45940)
Deprecate the pidfile setting #45938
Add node.processors setting in favor of processors #45855
Deprecate setting processors to more than available #44889

Infra/Settings

Add deprecation check for processors #45925 (issues: #45855, #45905)

Machine Learning

Only emit deprecation warning if there was actual change of a datafeed’s job_id. #44755
Deprecate the ability to update datafeed’s job_id. #44691 (issue: #44615)

New features

edit

Aggregations

Support Range Fields in Histogram and Date Histogram #45395
Add Cumulative Cardinality agg (and Data Science plugin) #43661 (issue: #43550)

Analysis

Add support for inlined user dictionary in the Kuromoji plugin #45489 (issue: #25343)

Authentication

PKI realm authentication delegation #45906 (issue: #34396)
PKI Authentication Delegation in new endpoint #43796 (issue: #34396)

Authorization

Add granular privileges for API keys #42020

Features/ILM

Add Snapshot Lifecycle Management #43934 (issue: #38461)

Features/Watcher

Add max_iterations configuration to watcher action with foreach execution #45715 (issues: #41997, #45169)

Geo

[SPATIAL] New ShapeQueryBuilder for querying indexed cartesian geometry #45108 (issue: #44980)
[GEO] New ShapeFieldMapper for indexing cartesian geometries #44980
Add Circle Processor #43851 (issue: #43554)
New shape field type for indexing Cartesian Geometries #43644

Machine Learning

Allow the user to specify query in Evaluate Data Frame request #45775 (issue: #45729)
Call the new _estimate_memory_usage API endpoint on data frame analytics _start #45536 (issues: #44699, #45544)
HLRC for memory usage estimation API #45531 (issues: #44699, #45188)
Implement ml/data_frame/analytics/_estimate_memory_usage API endpoint #45188 (issue: #44699)

Enhancements

edit

Aggregations

Add more flexibility to MovingFunction window alignment #44360 (issue: #42181)
Optimize Min and Max BKD optimizations #44315 (issue: #44290)
Allow pipeline aggs to select specific buckets from multi-bucket aggs #44179

Allocation

Defer reroute when starting shards #44433 (issues: #42105, #42738)
Allow RerouteService to reroute at lower priority #44338
Auto-release of read-only-allow-delete block when disk utilization fa… #42559 (issue: #39334)

Analysis

Allow all token/char filters in normalizers #43803 (issue: #43758)

Authentication

Allow Transport Actions to indicate authN realm #45767 (issue: #45331)
Explicitly fail if a realm only exists in keystore #44471 (issue: #44207)

Authorization

Add manage_own_api_key cluster privilege #45897 (issue: #40031)
Consider owner flag when retrieving/invalidating keys with API key service #45421 (issue: #40031)
REST API changes for manage-own-api-key privilege #44936 (issue: #40031)
Simplify API key service API #44935 (issue: #40031)

CCR

Include leases in error message when operations no longer available #45681

CRUD

Return seq_no and primary_term for noop update #44603 (issue: #42497)

Cluster Coordination

Improve slow logging in MasterService #45086 (issue: #45007)
More logging for slow cluster state application #45007
Ignore unknown fields if overriding node metadata #44689
Allow pending tasks before state recovery #44685 (issue: #44652)

Distributed

Do not create engine under IndexShard#mutex #45263 (issue: #43699)

Docs Infrastructure

add clarification around TESTSETUP docu and error message #43306

Engine

Flush engine after big merge #46066
Do sync before closeIntoReader when rolling generation to improve index performance #45765 (issue: #45371)
Refactor index engines to manage readers instead of searchers #43860
Async IO Processor release before notify #43682
Enable indexing optimization using sequence numbers on replicas #43616 (issue: #34099)

Features/ILM

Add node setting for disabling SLM #46794 (issue: #38461)
Include in-progress snapshot for a policy with get SLM policy API #45245
Add option to filter ILM explain response #44777 (issue: #44189)
Expose index age in ILM explain output #44457 (issue: #38988)

Features/Indices APIs

Add Clone Index API #44267 (issue: #44128)
Add description to force-merge tasks #41365 (issue: #15975)

Features/Ingest

Fix IngestService to respect original document content type #45799
Ingest Attachment: Upgrade tika to v1.22 #45575
Retrieve processors instead of checking existence #45354
Add ingest processor existence helper method #45156
Change the ingest simulate api to not include dropped documents #44161 (issue: #36150)

Features/Java High Level REST Client

Add XContentType as parameter to HLRC ART#createServerTestInstance #46036 (issue: #45970)
Add CloseIndexResponse to HLRC #44349 (issue: #39687)
Add mapper-extras and the RankFeatureQuery in the hlrc #43713 (issue: #43634)
removing background state update of Request object by RequestConverte… #40156 (issue: #39666)
Add delete aliases API to the high-level REST client #32909 (issue: #27205)

Features/Watcher

Add SSL/TLS settings for watcher email #45272 (issue: #30307)
Watcher reporting: add email warning if CSV attachment contains values that may be interperted as formulas #44460
Watcher add stopped listener #43939 (issue: #42409)
Improve CryptoService error message on missing secure file #43623 (issue: #43619)

Geo

Support WKT point conversion to geo_point type #44107 (issue: #41821)

Infra/Circuit Breakers

Fix G1 GC default IHOP #46169

Infra/Core

Add OCI annotations and adjust existing annotations #45167 (issues: #45162, #45166)
Use the full hash in build info #45163 (issue: #45162)

Infra/Packaging

Remove redundant Java check from Sys V init #45793 (issue: #45593)
Notify systemd when Elasticsearch is ready #44673

Infra/Plugins

Make plugin verification FIPS 140 compliant #44224 (issue: #41263)

Infra/Scripting

Whitelist randomUUID in Painless #45148 (issue: #39080)
Add missing ZonedDateTime methods for joda compat layer #44829 (issue: #44411)
Remove stale permissions from untrusted policy #44783

Infra/Settings

Add more meaningful keystore version mismatch errors #46291 (issue: #44624)
Lift the restrictions that uppercase is not allowed in Setting Name. #45222 (issue: #43835)
Normalize environment paths #45179 (issue: #45176)

Machine Learning

Support boolean fields for data frame analytics #46037
Add description to data frame analytics #45774
Add regression analysis to data frame analytics #45292
Introduce formal node ML role #45174 (issues: #29943, #43175)
Improve CSV header row detection in find_file_structure #45099 (issue: #45047)
Outlier detection should only fetch docs that have the analyzed fields #44944
Persist DatafeedTimingStats with RefreshPolicy.NONE by default #44940 (issue: #44792)
Add result_type field to TimingStats and DatafeedTimingStats documents #44812
Implement exponential average search time per hour statistics. #44683 (issue: #29857)
Add r_squared eval metric to regression #44248
Adds support for regression.mean_squared_error to eval API #44140
Add DatafeedTimingStats.average_search_time_per_bucket_ms and TimingStats.total_bucket_processing_time_ms stats #44125 (issue: #29857)
Add DatafeedTimingStats to datafeed GetDatafeedStatsAction.Response #43045 (issue: #29857)

Network

Better logging for TLS message on non-secure transport channel #45835 (issue: #32688)
Asynchronously connect to remote clusters #44825 (issue: #40150)
Improve errors when TLS files cannot be read #44787 (issue: #43079)
Add per-socket keepalive options #44055
Move ConnectionManager to async APIs #42636

Ranking

Search enhancement: pinned queries #44345 (issue: #44074)
Fix parameter value for calling data.advanceExact #44205
Add l1norm and l2norm distances for vectors #44116 (issue: #37947)

Recovery

Ignore translog retention policy if soft-deletes enabled #45473 (issue: #45136)
Only retain reasonable history for peer recoveries #45208 (issue: #41536)
Use index for peer recovery instead of translog #45136 (issues: #38904, #41536, #42211)
Trim local translog in peer recovery #44756
Make peer recovery send file chunks async #44468 (issues: #36195, #44040)

SQL

SQL: Support queries with HAVING over SELECT #46709 (issue: #37051)
SQL: Break TextFormatter/Cursor dependency #45613 (issue: #45516)
SQL: remove deprecated use of "interval" from date_histogram usage #45501 (issue: #43922)
SQL: use hasValue() methods from Elasticsearch’s InspectionHelper classes #44745 (issue: #35745)
Switch from using docvalue_fields to extracting values from _source #44062 (issue: #41852)

Search

Adds usage stats for vectors: #44512
Associate sub-requests to their parent task in multi search API #44492
Cancel search task on connection close #43332

Security

Set security index refresh interval to 1s #45434 (issue: #44934)

Snapshot/Restore

add disable_chunked_encoding configuration #44052
Repository Cleanup Endpoint #43900

Task Management

Remove task null check in TransportAction #45014
TaskListener#onFailure to accept Exception instead of Throwable #44946
Move child task cancellation to TaskManager #44573 (issue: #44494)

Transforms

Add update transform api endpoint #45154 (issue: #43438)
Add support for bucket_selector #44718 (issues: #43744, #44557)
Add force delete #44590 (issue: #43961)
Add dynamic cluster setting for failure retries #44577
Add optional defer_validation param to PUT #44455 (issue: #43439)
Add support for geo_bounds aggregation #44441
Add a frequency option to transform config, default 1m #44120

Bug fixes

edit

Aggregations

Fix early termination of aggregators that run with breadth-first mode #44963 (issue: #44909)
Support BucketScript paths of type string and array. #44694 (issue: #44385)

Allocation

Avoid overshooting watermarks during relocation #46079 (issue: #45177)
Cluster health should await events plus other things #44348
Do not copy initial recovery filter during split #44053 (issue: #43955)

Analysis

Enable reloading of synonym_graph filters #45135 (issue: #45127)
Fix AnalyzeAction response serialization #44284 (issue: #44078)

Authentication

Fallback to realm authc if ApiKey fails #46538
Enforce realm name uniqueness #46253
Allow empty token endpoint for implicit flow #45038
Do not use scroll when finding duplicate API key #45026
Fix broken short-circuit in getUnlicensedRealms #44399
Fix X509AuthenticationToken principal #43932 (issues: #34396, #43796)

Authorization

Do not rewrite aliases on remove-index from aliases requests #46989
Give kibana user privileges to create APM agent config index #46765 (issue: #45610)
Add manage_own_api_key cluster privilege #45696 (issue: #40031)
Sparse role queries can throw an NPE #45053

CCR

Clean up ShardFollowTasks for deleted indices #44702 (issue: #34404)
Skip update if leader and follower settings identical #44535 (issue: #44521)
Avoid stack overflow in auto-follow coordinator #44421 (issue: #43251)
Avoid NPE when checking for CCR index privileges #44397 (issue: #44172)

CRUD

Ignore replication for noop updates #46458 (issues: #41065, #44603, #46366)
Allow _update on write alias #45318 (issue: #31520)
Do not allow version in Rest Update API #43516 (issue: #42497)

Cluster Coordination

Assert no exceptions during state application #47090 (issue: #47038)
Avoid counting votes from master-ineligible nodes #43688

Distributed

Fix false positive out of sync warning in synced-flush #46576 (issues: #28464, #30244)
Suppress warning logs from background sync on relocated primary #46247 (issues: #40800, #42241)
Ensure AsyncTask#isScheduled remain false after close #45687 (issue: #45576)
Update translog checkpoint after marking operations as persisted #45634 (issue: #29161)
Fix clock used in update requests #45262 (issue: #45254)
Restore DefaultShardOperationFailedException’s reason during deserialization #45203
Use IndicesModule named writables in elasticsearch-shard tool #45036 (issue: #44628)

Engine

Handle delete document level failures #46100 (issue: #46083)
Handle no-op document level failures #46083
Remove leniency during replay translog in peer recovery #44989
Throw TranslogCorruptedException in more cases #44217
Fail engine if hit document failure on replicas #43523 (issues: #40435, #43228)

Features/ILM

Handle partial failure retrieving segments in SegmentCountStep #46556
Fixes for API specification #46522
Fix SnapshotLifecycleMetadata xcontent serialization #46500 (issue: #46499)
Updated slm API spec parameters and URL #44797
Fix swapped variables in error message #44300

Features/Indices APIs

Fix a bug where mappings are dropped from rollover requests. #45411 (issue: #45399)
Create index with typeless mapping #45120
Check shard limit after applying index templates #44619 (issues: #34021, #44567, #44619)
Validate index settings after applying templates #44612 (issues: #34021, #44567)

Features/Ingest

Allow dropping documents with auto-generated ID #46773 (issue: #46678)

Features/Java High Level REST Client

[HLRC] Send min_score as query string parameter to the count API #46829 (issue: #46474)
HLRC multisearchTemplate forgot params #46492 (issue: #46488)
terminateAfter added to the RequestConverter #46474 (issue: #46446)
[Closes #44045] Added slices parameter when submitting reindex request via Java high level REST client #45690 (issue: #44045)
HLRC: Fix + Not Correctly Encoded in GET Req. #33164 (issue: #33077)

Features/Watcher

Fix class used to initialize logger in Watcher #46467
Fix wrong URL encoding in watcher HTTP client #45894 (issue: #44970)
Fix watcher HttpClient URL creation #45207 (issue: #44970)
Log write failures for watcher history document. #44129

Geo

Geo: fix geo query decomposition #44924 (issue: #44891)
Geo: add validator that only checks altitude #43893

Highlighting

Fix highlighting for script_score query #46507 (issue: #46471)

Infra/Core

Always check that cgroup data is present #45606 (issue: #45396)
Safe publication of DelayedAllocationService and SnapshotShardsService #45517 (issue: #38560)
Add default CLI JVM options #44545 (issues: #219, #42021)
Fix decimal point parsing for date_optional_time #43859 (issue: #43730)

Infra/Logging

Fix types field in JSON Search Slow Logs #44641
Add types field to JSON slow logs in 7.x #44592 (issues: #41354, #44178)

Infra/Packaging

Add destructiveDistroTest meta task #45762
Use bundled JDK in Sys V init #45593 (issue: #45542)
Restore setting up temp dir for windows service #44541

Infra/Plugins

Do not checksum all bytes at once in plugin install #44649 (issue: #44545)

Infra/REST API

Improve error message when index settings are not a map #45588 (issue: #45126)
Add is_write_index column to cat.aliases #44772
Fix URL documentation in API specs #44487

Infra/Scripting

Fix bugs in Painless SCatch node #45880
Fix JodaCompatibleZonedDateTime casts in Painless #44874

Infra/Settings

bug fix about elasticsearch.common.settings.Settings.processSetting #44047 (issue: #43791)

Machine Learning

Fix two datafeed flush lockup bugs #46982
Throw an error when a datafeed needs CCS but it is not enabled for the node #46044 (issue: #46025)
Handle "null" value of Estimate memory usage API response gracefully. #45726 (issue: #44699)
Remove timeout on waiting for data frame analytics result processor to complete #45724 (issue: #45723)
Check dest index is empty when starting data frame analytics #45094
Catch any error thrown while closing data frame analytics process #44958
Treat PostDataActionResponse.DataCounts.bucketCount as incremental rather than absolute (total). #44803 (issue: #44792)
Treat big changes in searchCount as significant and persist the document after such changes #44413 (issues: #44196, #44335)
Update .ml-config mappings before indexing job, datafeed or data frame analytics config #44216 (issue: #44263)
Wait for .ml-config primary before assigning persistent tasks #44170 (issue: #44156)
Fix ML memory tracker lockup when inner step fails #44158 (issue: #44156)
Fix datafeed checks when a concrete remote index is present #43923 (issue: #42113)
Rename outlier detection method values knn and tnn to distance_kth_nn and distance_knn respectively to match the API. #598
Fix occasional (non-deterministic) reinitialisation of modeling for the lat_long function. #641

Mapping

Make sure to validate the type before attempting to merge a new mapping. #45157 (issues: #29316, #43012)
Ensure field caps doesn’t error on rank feature fields. #44370 (issue: #44330)
Prevent types deprecation warning for indices.exists requests #43963 (issue: #43905)
Add include_type_name in indices.exitst REST API spec #43910 (issue: #43905)

Network

Fix Broken HTTP Request Breaking Channel Closing #45958 (issues: #43362, #43850)
Fix plaintext on TLS port logging #45852 (issue: #32688)
transport.publish_address should contain CNAME #45626 (issues: #32806, #39970)
Fix bug in copying bytes for socket write #45463 (issue: #45444)

Recovery

Never release store using CancellableThreads #45409 (issues: #45136, #45237)
Remove leniency in reset engine from translog #44711

Rollup

Fix Rollup job creation to work with templates #43943

SQL

SQL: Properly handle indices with no/empty mapping #46775 (issue: #46757)
SQL: improve ResultSet behavior when no rows are available #46753 (issue: #46750)
SQL: use the correct data type for types conversion #46574 (issue: #46090)
SQL: Fix issue with common type resolution #46565 (issue: #46551)
SQL: fix scripting for grouped by datetime functions #46421 (issue: #40241)
SQL: Use null schema response #46386 (issue: #46381)
SQL: Fix issue with IIF function when condition folds #46290 (issue: #46268)
SQL: Fix issue with DataType for CASE with NULL #46173 (issue: #46032)
SQL: adds format parameter to range queries for constant date comparisons #45326 (issue: #45139)
SQL: uniquely named inner_hits sections for each nested field condition #45039 (issues: #33080, #44544)
SQL: fix URI path being lost in case of hosted ES scenario #44776 (issue: #44721)
SQL: change the size of the list of concrete indices when resolving multiple indices #43878 (issue: #43876)
SQL: handle double quotes escaping #43829 (issue: #43810)
SQL: add pretty printing to JSON format #43756
SQL: handle SQL not being available in a more graceful way #43665 (issue: #41279)

Search

Multi-get requests should wait for search active #46283 (issue: #27500)
Ensure top docs optimization is fully disabled for queries with unbounded max scores. #46105 (issue: #45933)
Disallow partial results when shard unavailable #45739 (issue: #42612)
Prevent Leaking Search Tasks on Exceptions in FetchSearchPhase and DfsQueryPhase #45500
Fix an NPE when requesting inner hits and _source is disabled. #44836 (issue: #43517)
Don’t use index_phrases on graph queries #44340 (issue: #43976)

Security

Initialize document subset bit set cache used for DLS #46211 (issue: #45147)
Fix .security-* indices auto-create #44918
SecurityIndexManager handle RuntimeException while reading mapping #44409
Do not swallow I/O exception getting authentication #44398 (issues: #44172, #44397)
Use system context for looking up connected nodes #43991 (issue: #43974)

Snapshot/Restore

Fix Bug in Snapshot Status Response Timestamps #46919 (issue: #46913)
GCS deleteBlobsIgnoringIfNotExists should catch StorageException #46832 (issue: #46772)
Fix TransportSnapshotsStatusAction ThreadPool Use #45824
Stop Executing SLM Policy Transport Action on Snapshot Pool #45727 (issue: #45594)
Check again on-going snapshots/restores of indices before closing #43873
Make Timestamps Returned by Snapshot APIs Consistent #43148 (issue: #43074)
Recursively Delete Unreferenced Index Directories #42189 (issue: #13159)

Task Management

Catch AllocatedTask registration failures #45300

Transforms

Use field_caps API for mapping deduction #46703 (issue: #46694)
Fix off-by-one error in checkpoint operations_behind #46235
Moves failure state transition for MT safety #45676 (issue: #45664)
Fix _start?force=true bug #45660
Fix failure state transitions and race condition #45627 (issues: #45562, #45609)
Fix starting a batch data frame after stopping at runtime #45340 (issues: #44219, #45339)
Fix null aggregation handling in indexer #45061 (issue: #44906)
Unify validation exceptions between PUT and _preview #44983 (issue: #44953)
Treat bulk index failures as an indexing failure #44351 (issue: #44101)
Prevent task from attempting to run when failed #44239 (issue: #44121)
Respond with 409 status code when failing _stop #44231 (issue: #44103)
Add index validations to _start data frame transform #44191 (issue: #44104)
Data frame task failure does not make a 500 response #44058 (issue: #44011)
Audit message missing for autostop #43984 (issue: #43977)