IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Elasticsearch version 7.12.1 Elasticsearch version 7.11.2 »
Elastic Docs ›Elasticsearch Guide [7.12] ›Release notes

Elasticsearch version 7.12.0

edit

Elasticsearch version 7.12.0

edit

Also see Breaking changes in 7.12.

Security updates

edit
  • A document disclosure flaw was found in the Elasticsearch suggester and profile API when Document and Field Level Security are enabled. The suggester and profile API are normally disabled for an index when document level security is enabled on the index. Certain queries are able to enable the profiler and suggester which could lead to disclosing the existence of documents and fields the attacker should not be able to view. All versions of Elasticsearch before 6.8.15 and 7.11.2 are affected by this flaw. You must upgrade to 6.8.15 or 7.11.2 to obtain the fix. CVE-2021-22137

Known issues

edit

  • Frozen tier: (Windows only) The frozen data tier relies on multiple caching mechanisms to speed up access to searchable snapshot files. One of these caches uses sparse files to avoid creating large files on disk when it is not strictly required. A bug prevented files from being created with the right options to enable sparse support on Windows, leading Elasticsearch to create potentially large files that can end up consuming all the disk space.

    This issue is fixed in Elasticsearch versions 7.15.2 and later. There is no known workaround for earlier versions. Filesystems that enable sparse files by default are not affected. For more details, see #79371.

  • If autoscaling is enabled for machine learning, the administrator of the cluster should increase the cluster setting xpack.ml.max_open_jobs. This allows autoscaling to run reliably as it relies on assigning jobs only via memory. Having xpack.ml.max_open_jobs as a small number may cause autoscaling to behave unexpectedly.
  • If autoscaling is enabled and job assignment takes an inordinate amount of time, scaling sizes may increase much higher than required. Elasticsearch 7.13.0 contains a fix for this. For more details, see #72509

  • Snapshot and restore: If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail. To mitigate this problem, prevent concurrent snapshot operations by setting snapshot.max_concurrent_operations: 1.

    This issue is fixed in Elasticsearch versions 7.13.1 and later. For more details, see #73456.

Breaking changes

edit
Mapping
  • Returning nested data in fields API #67432 (issue: #63709)
Query Languages
SQL
  • Test and fix the NULL handling of the string functions #68379 (issue: #58907)

Breaking Java changes

edit
Features/Java High Level REST Client
  • Move find file structure finder in REST High Level client to its new endpoint and plugin #67290

Deprecations

edit
Distributed
Infra/Core
  • Add deprecation warning for removed strict_duplicate_detection setting #63500 (issues: #34588, #60222)
Geo
  • Deprecate GeoPolygon query in favour of GeoShape query #64227 (issue: #48928)
Machine Learning
  • Add deprecation check for job model snapshots that need upgrade #66062 (issue: #64154)
Security
  • Deprecate the id field for the InvalidateApiKey API #66317 (issue: #63224)

New features

edit
Authorization
  • Add a cluster privilege to cancel tasks and delete async searches #68679 (issue: #67965)
EQL
  • Introduce ~ grammar for case-insensitive functions #67869 (issue: #67868)
  • Introduce like and regex keywords #68791 (issue: #68639)
Features/ILM+SLM
  • Add support for partial searchable snapshots to ILM #68714 (issues: #68509, #68605)
  • Add the frozen tier node role and ILM phase #68605 (issue: #60848)
  • Support max_single_primary_size in resize action and expose in ILM #67705 (issue: #65714)
Features/Ingest
  • Fingerprint ingest processor #68415 (issue: #53578)
  • Network community_id processor for ingest pipelines #66534 (issue: #55685)
  • Network direction processor #66644
Query Languages
  • Implement support for date_nanos Elasticsearch field type #67666 (issue: #38562)
SQL
Search
  • Add minimum compatibility version to SearchRequest #65896 (issue: #63304)
  • Sort field tiebreaker for point in time (PIT) readers #66093 (issue: #56828)
Snapshot/Restore
  • Add partial searchable snapshot support for a frozen tier #68509

Enhancements

edit
Aggregations
Allocation
  • Drop gateway.recover_after_nodes from default conf #68000 (issue: #53646)
  • Make InternalClusterInfoService async #66993
Authentication
  • Add grant-api-key to HLRC #68190
Authorization
  • Add more context to cluster access denied messages #66900 (issues: #42166, #60357)
  • Add more context to index access denied errors #60357 (issue: #42166)
  • Allow Kibana to create Fleet Server indices #68152
  • Phase 2 support for operator privileges: Cluster settings #66684
  • view_index_metadata and manage privs now grant access to field capabilities #67392 (issue: #66867)
Cluster Coordination
  • Clarify message emitted on cluster UUID mismatch #66915
  • Extend default probe connect/handshake timeouts #68059 (issue: #42636)
  • Skip cluster state serialization to closed channel #67413
EQL
  • Add result_position to correctness test #66761
  • Introduce EQL search status API #68065 (issue: #66955)
Engine
  • Assign id to searcher using ids of segments #66668 (issue: #63963)
  • Use merging fieldsreader when restoring versionmap during recovery #66944
Features/ILM+SLM
  • Make the rest of the forcemerge action steps retryable #66352 (issue: #48183)
  • Make the unfollow action and CCR related steps retryable #66356 (issue: #48183)
  • Skip unfollow action if the index is not a follower #68690
Features/Indices APIs
  • Add max_single_primary_size as a condition for the rollover index API #67842 (issue: #63026)
  • Make template conflict warning deterministic #67066 (issue: #66820)
Features/Ingest
  • Configurable MIME type for mustache template encoding on set processor #65314 (issue: #65115)
Features/Java High Level REST Client
  • Added support for passing http query parameters #67802 (issue: #53846)
Features/Java Low Level REST Client
  • Add client metadata header on RestClient requests #66303 (issue: #66189)
Features/Stats
  • Add processor architectures to cluster stats #68264
  • Make GET _cluster/stats cancellable #68676 (issue: #55550)
Geo
  • Add support for Spatial Relationships to geo_point field #67631 (issue: #52382)
  • Implement IndexOrDocValuesQuery for geo_shape field #64688
Highlighting
  • Add query param to limit highlighting to specified length #67325 (issue: #52155)
Infra/Core
  • Clarify bootstrap check failure messages #67501
  • Introduce "Feature States" for managing snapshots of system indices #63513 (issue: #61657)
Infra/Logging
Infra/Plugins
  • Notify users to restart Elasticsearch after plugin installation #66723 (issue: #39487)
Infra/Scripting
  • Add OSS whitelist to execute API #67038 (issue: #67035)
  • Capture structured javadoc from stdlib #68782
  • Enforce GPLv2 for parsed stdlib docs #68601
  • Improve bad regex pattern syntax error #68520
  • Parse stdlib files for parameter names #67837
  • Whitelist JSON functions for ingest #67118
Infra/Settings
  • Add enumsetting to org.elasticsearch.common.settings.Setting #66826 (issue: #65335)
  • Support removing archived settings if cluster has read only blocks #64113
License
  • Remove grace period from license expiration check #67316
Machine Learning
  • Add new multi custom processor for data frame analytics and model inference #67362
  • Allow data frame analytics memory estimation on non-ML nodes #68146 (issue: #67840)
  • Automatic management for machine learning system indices #68044
  • Expand regression and classification hyperparameters #67950
  • Improve resuming a data frame analytics job stopped during inference #67623
  • Move find file structure to a new API endpoint #67123 (issue: #67001)
  • Preserve inference progress when starting data frame analytics #68222
  • Write notification messages indicating required capacity when a job is not assigned to a node #67181
  • Fix edge case which could cause spurious anomalies early in the learning process if the time series has non-diurnal seasonality #1634
  • Compute importance of hyperparameters optimized in the fine parameter tuning step #1627
  • Early stopping for the fine parameter tuning step of classification and regression model training #1676
  • Correct upgrade for pre-6.3 state for lat_long anomaly anomaly detectors #1681
  • Per tree feature bag to speed up training of regression and classification models and improve scalability for large numbers of features #1733
Mapping
  • Shortcut to avoid fmod #66909
Packaging
  • Tweaks to the Iron Bank docker context #66942
Performance
  • Add benchmark racing scripts #68369
Query Languages
  • Adapt nested fields extraction from fields API output to the new un-flattened structure #68745 (issue: #68722)
  • Simplify arithmetic operations in binary comps #66022 (issue: #65394)
Ranking
  • Add linear function to rank_feature query #67438 (issue: #49859)
Rollup
  • Remove the rollup thread pool #65958
SQL
  • Enhance error message on filtering check against aggs #68763 (issue: #57125)
Search
  • Add grok and dissect methods to runtime fields #68088 (issue: #67825)
  • Allow deletion of async searches with the manage privilege #67965
  • Avoid duplicate serialization for TermsQueryBuilder #67223 (issue: #67132)
  • Early detection of circuit breaker exception in the coordinating node #67431 (issue: #62884)
  • Limit the depth of nested bool queries #66204 (issue: #55303)
  • Remove unnecessary optimizations for TermsSetQueryBuilder #67637 (issue: #67223)
  • Retry point in time on other copy when possible #66713 (issue: #61062)
  • Slightly speed up scripts #68478
  • Use a mapping snapshot for fetching nested docs #66877 (issue: #66295)
Security
  • Build complex automatons more efficiently #66724 (issue: #36062)
Snapshot/Restore
  • Add ClusterUUID to RepositoryData #68002
  • Allow searchable snapshots to work on source-only repositories #67975 (issue: #67936)
  • Avoid async cache-size fetch on partial shards #68644
  • Forbid removing write block from searchable snapshot index #68036
  • Ignore disk watermarks on partial shards #68673
  • Improve error message on incompatible repo format #67776
  • Introduce repository UUIDs #67829 (issue: #66431)
  • Introduce repository test kit/analyser #67247
  • Permit rename of repository of searchable snapshot #67968 (issue: #66431)
  • Reject remounting snapshot of a searchable snapshot #68816 (issue: #68792)
Task Management
Transform
  • Add support for search-time runtime fields #67643 (issue: #65147)
  • Automatic management for transform system indices #68590
  • Implement retention policy to delete data from a transform #67832 (issue: #67916)
  • Refactor cat transform to show more useful information #68232 (issues: #66367, #66718)
  • Report last search time in transform stats #66718 (issues: #66367, #66410, #66718)

Bug fixes

edit
Aggregations
  • Remove toString->parse roundtrip for missing value in terms agg #67954 (issue: #67197)
Engine
  • Fail force-merges on read-only engines #64756
Features/Features
  • Prevent unnecessary system index access warnings in Deprecation Info API #67312 (issue: #66063)
Features/ILM+SLM
  • Avoid exception in in SetStepInfoUpdateTask#onFailure #67553
  • Fix issues with CS Handling in ILM async steps #68361
Features/Java Low Level REST Client
  • Fix wrong totalIterationCount when running BulkIndexBenchmark #64515 (issue: #61451)
Features/Monitoring
Geo
  • Fix indexing of degenerated rectangles #67702 (issue: #67695)
Infra/Core
  • Allow the *,-* ("no-index") pattern for destructive actions when destructive_requires_name is true #68021 (issue: #67958)
Infra/Scripting
  • Augmentation.join can’t handle empty strings at the start #68251 (issue: #33434)
Machine Learning
  • Make find_structure validate lines_to_sample at least 2 #68199 (issue: #68128)
  • Return 400 status for invalid charset in find_structure #68201 (issue: #68130)
  • Return status 400 for grok_pattern errors in find_structure #68205 (issue: #68132)
  • Fix a source of instability in time series modeling for anomaly detection. This has been observed to cause spurious anomalies for a partition which no longer receives any data #1675
  • Ensure that we stop modeling seasonality for data which flatlines. This is important for count and sum detectors which treat empty buckets as zero. Before this change, we could sometimes detect spurious anomalies in realtime detection after a partition no longer received any data #1654
SQL
  • Fix ZonedDateTime with nanos serialisation #68253 (issue: #67666)
Search
  • Handle ignored fields directly in SourceValueFetcher #68738
Snapshot/Restore
  • Make RepositoryData Parsing Stricter #67699 (issue: #67696)
« Elasticsearch version 7.12.1 Elasticsearch version 7.11.2 »