Elasticsearch version 7.14.0

edit

Elasticsearch version 7.14.0

edit

Also see Breaking changes in 7.14.

Known issues

edit
  • Frozen tier: (Windows only) The frozen data tier relies on multiple caching mechanisms to speed up access to searchable snapshot files. One of these caches uses sparse files to avoid creating large files on disk when it is not strictly required. A bug prevented files from being created with the right options to enable sparse support on Windows, leading Elasticsearch to create potentially large files that can end up consuming all the disk space.

    This issue is fixed in Elasticsearch versions 7.15.2 and later. There is no known workaround for earlier versions. Filesystems that enable sparse files by default are not affected. For more details, see #79371.

  • Aggregations: In Elasticsearch 7.14.0–7.15.0, when a cross-cluster search (CCS) request is proxied, the memory for the aggregations on the proxy node will not be freed. The trigger is cross-cluster search using aggregations where minimize roundtrips is not effective (for example, when minimize roundtrips is explicitly disabled, or implicitly disabled when using scroll, async and point-in-time searches).

    This affects Kibana CCS aggregations because Kibana uses async search by default. This issue can also happen in all modes of remote connections configured for cross-cluster search (sniff and proxy). In sniff mode, we only connect to a subset of the remote nodes (by default 3). So if the remote node we want to send a request to is not one of those 3, we must send the request as a proxy request. The workaround is to periodically restart nodes with heap pressure.

    We have fixed this issue in Elasticsearch 7.15.1 and later versions. For more details, see #78404.

  • Snapshot and restore: If a running snapshot is cancelled while a previously-started snapshot is still ongoing and a later snapshot is enqueued then there is a risk that some shard data may be lost from the repository. This will cause future restore operations to fail. To mitigate this problem, set snapshot.max_concurrent_operations: 1 to prevent concurrent snapshot operations:

    PUT _cluster/settings
    {
      "persistent" : {
        "snapshot.max_concurrent_operations" : 1
      }
    }

    This issue is fixed in Elasticsearch versions 7.14.1 and later. It is not possible to repair a repository once it is affected by this issue, so you must restore the repository from a backup, or clear the repository by executing DELETE _snapshot/<repository name>/*, or move to a fresh repository. For more details, see #75598.

  • Parsing a request when the last element in an array is filtered out (for instance using _source_includes) fails. This is due to a bug in Jackson parser. Fixed in Elasticsearch 8.6.1 (#91456)
  • The deprecated index.mapper.dynamic setting can break your cluster. It can only be set using the Update index settings API. Symptoms include nodes failing to start or shards failing to allocate. Do not use this setting in versions prior to 7.17.22. The bug is fixed in 7.17.22. (issue: #109160)

Breaking changes

edit
CCR
  • Deprecate auto-follow system indices #73237 (issue: #72815)
Cluster Coordination
  • Extend version barrier to all upgrades #73358 (issue: #72911)

Deprecations

edit
Allocation
  • Add entry to deprecation API for CLUSTER_ROUTING_ALLOCATION_INCLUDE_RELOCATIONS_SETTING #73552 (issue: #47717)
  • Deprecations for single data node setting #73733 (issue: #55805)
Analysis
  • Deprecate setting version on analyzers #74073
Authentication
  • Deprecate realm names with a leading underscore #73366 (issue: #73250)
Authorization
  • Designate reporting_user as a deprecated role #72057
Features/Indices APIs
Infra/Core
  • Add deprecation for single valued list of data path #72180
  • Add warning for path.data as a list to deprecation api #72183 (issue: #72180)
Infra/Logging
  • Deprecate slow log level #73718
Mapping
  • Add deprecation warnings to geo fields that define multifields #71912
Security
  • Deprecate camelCase parameters used by SAML APIs #73984 (issue: #73543)
  • Deprecate security implicitly disabled on trial/basic #72339

New features

edit
Features/Data streams
  • Add basic alias support for data streams #72613 (issue: #66163)
Features/ILM+SLM
  • Service to migrate indices and ILM policies to data tiers #73689 (issue: #73154)
Mapping
  • Add match_only_text, a space-efficient variant of text #66172 (issue: #64467)
SQL
  • Introduce an async querying mode for SQL #73991 (issue: #71041)
Search
  • New terms_enum API for discovering terms in the index #66452

Enhancements

edit
Aggregations
  • Add keep_values gap policy #73297 (issue: #27377)
  • Add extra profiling information to terms agg #73636
  • Add support for range aggregations on histogram mapped fields #74146 (issue: #74071)
  • Only enable "filter by filter" optimization for terms, date_histogram, range, and fitlers aggregation when we’re sure it’ll be faster #74260
  • Docvalueformat errors #73121 (issue: #72040)
  • Optimize lone single bucket date_histogram #71180 (issue: #69377)
  • Read aggregations directly from pooled buffers #72309
  • Reduce agg buckets only if competitive #74096
Allocation
  • Log cluster health change when node leaves #73576 (issue: #68948)
Audit
  • Service Accounts - add service token type to audit log #73399 (issues: #72198, #73135)
  • Service Accounts - audit for security config change #72555 (issue: #72198)
  • Service Accounts - audit logging for service token name #72198
Authentication
  • Add more logging to SAML Realm #73279
  • Service Accounts - Add token source to authenticate metadata #73135
  • Service Accounts - enforcing token secret min length at authc time #72519
  • ApiKeyAuthCache now expires after access instead of write #73982
Authorization
  • Granting kibana_system reserved role access to "all" privileges to .alerts* and .siem-signals* index #72181
  • Improve error when handling unexpected wildcards #73079 (issues: #43015, #61245)
  • Make RBACEngine.resolveAuthorizedIndicesFromRole Return Set for Lookups #72598
  • Service Accounts - Disallow oauth2 creation #72553
  • Service Accounts - no roles in denial error message #72517
  • Support shard request cache for queries with DLS and FLS #70191 (issue: #44581)
CCR
  • Add support for index pattern exclusion in CCR AutoFollow #72935 (issue: #67686)
Client
  • Service Accounts - HLRC #72431
Cluster Coordination
  • Increase PeerFinder verbosity on persistent failure #73128 (issue: #72968)
  • Log elapsed time for host resolution timeouts #72484
Distributed
  • Run CheckIndex on metadata index before loading #73239 (issue: #29358)
Features/Features
  • Add X-Elastic-Product header on all HTTP responses #73434 (issue: #73424)
  • Add meta field to deprecation issue definition #74085 (issue: #73089)
  • Better out-of-the-box mappings for logs, metrics and synthetics #64978 (issue: #64400)
Features/ILM+SLM
  • Add _meta field to ilm policy #73515 (issue: #70755)
  • Add _meta information to built-in ILM policies #73629
  • Make ILM Steps use Infinite Master Timeout #74143 (issue: #72085)
  • Retry ILM force merge step on shard failures #73762 (issue: #73142)
Features/Ingest
  • Add template snippets support for KV ingest processor #73758 (issue: #73377)
  • Enable GeoIP downloader by default #74323
  • Change GeoIP downloader policy after 30 days of no updates #74099
  • Expose Community ID processor in Painless #73963 (issue: #73346)
  • Expose URI parts processor in Painless #73634 (issue: #73346)
Features/Java High Level REST Client
  • Add point in time to HLRC #72167 (issue: #70593)
  • Verify that main info response returns correct product headers #73910 (issue: #73434)
  • Change GET method to POST in Searchtemplate #72851 (issues: #72073, #72397)
  • Adds new running_state field to datafeed stats (#73926) #74025
  • Adds support for reset job API #74254
Features/Stats
Features/Watcher
  • Fix Watcher HTTP connection config for longevity #72736 (issue: #52997)
Geo
  • Speed up geotile aggregation over geo_shape field #72984
  • Add painless script support for geo_shape field #72886 (issue: #54218)
Infra/Core
  • Add thread pool for critical operations on system indices #72625 (issue: #69251)
  • Add unenroll_timeout/unenrolled_reason field to Fleet system indices #74180
  • Add warnings if timer thread is late to wake up #72465
  • Log slow executions on scheduler thread #72606 (issue: #72465)
  • Protect newly introduced system indices fully #74186 (issue: #72572)
  • RestController not using thread context directly from thread pool #74293
Infra/Node Lifecycle
  • Don’t assign persistent tasks to nodes shutting down #72260 (issue: #70338)
  • Make ILM aware of node shutdown #73690 (issue: #70338)
Infra/Scripting
  • Delegate to wrapped map for toString in DynamicMap #72048
Machine Learning
  • Add new bucket_correlation aggregation with initial count_correlation function #72133
  • Add new ml_standard tokenizer for ML categorization #72744
  • Adding running_state to datafeed stats object #73926
  • Adding new KS test pipeline aggregation #73334
  • Adds latest record timestamp to model snapshot deprecation warning #73066
  • Closing an anomaly detection job now automatically stops its datafeed if necessary #74257
  • Further increase memory estimates for categorization #73370 (issue: #68859)
  • Make ml_standard tokenizer create single tokens for email addresses #73052
  • Make ml_standard tokenizer the default for new categorization jobs #72805 (issue: #1724)
  • Optimize inference step when there are no test docs #74315
  • Reset anomaly detection job API #73908
  • Switch ML internal index templates to composable templates #73232 (issue: #65437)
  • Give higher weight to multiple adjacent dictionary words when performing categorization #1903
Mapping
  • Add a cluster deprecation check for index templates containing multiple types #72540
  • Enhance error message for copy_to #72820 (issue: #49344)
  • Make intervals queries fully pluggable through field mappers #71429
  • DynamicFieldType to expose its known subfields names #73530
  • FieldTypeLookup to support dynamic runtime fields #73519
Packaging
  • Pin Alpine Linux version in Docker builds #74169
  • Tighten up write permissions in Docker image #73942 (issue: #70635)
Recovery
  • Fork the sending of file chunks during recovery #74164
  • Treat writtenBy as an opaque string #74125
Search
  • Add search_after support to new termsEnum API #72933 (issue: #72910)
  • Be cleverer about extracting nested sources #73157
  • Integrate circuit breaker in AsyncTaskIndexService #73862 (issues: #67594, #73638)
  • PIT: Make validation error actionable #74224 (issue: #74223)
  • Support field collapsing with search_after #73023 (issue: #53115)
  • Support much larger source filters #72277
  • Write async response directly to XContent to reduce memory usage #73707 (issue: #67594)
Security
  • Add REST API specification for SAML APIs #72839 (issue: #67189)
  • Cache API key hashing results on creation time #74106
  • Expose API Key metadata to SetSecurityUser ingest processor #72137 (issue: #71024)
  • Move hashing on API key creation to crypto thread pool #74165 (issue: #74106)
  • Service Accounts - show token name for name validation failures #73131 (issue: #73081)
Snapshot/Restore
  • Abort writes in repo analyzer #72077 (issue: #72051)
  • Add more snapshot details to repo data #72232
  • Allow some repository settings to be updated dynamically #72543
  • Cache RepositoryData outright instead of serialized #73190
  • Introduce next field in paginated GetSnapshots Response #74236 (issue: #73952)
  • Introduce SNAPSHOT_META thread pool for fetching repository metadata #73172
  • Make large bulk snapshot deletes more memory efficient #72788
  • Make GetSnapshotsAction cancellable #72644
  • Make SnapshotStatusAction cancellable #73818 (issue: #72644)
  • Pagination and sorting for get snapshots API #73952
  • Prevent deletion of repositories that are used by snapshot-backed indices #73714
  • Reroute when new repository is registered #73761 (issues: #73669, #73714)
  • Save memory on large repository metadata blob writes #74313
Task Management
  • Add support for Rest XPackUsage task cancellation #72304
  • Add support for RestGetMapping cancellation #72234
  • Add support for task cancellation to TransportMasterNodeAction #72157
  • Identify cancelled tasks in list tasks API #72931 (issue: #72907)
  • Log at DEBUG only on disconnect during cancellation #74042 (issues: #65443, #72968)
Transform
  • Add support for top metrics #71850 (issues: #51925, #52236)
  • Improve error message when user lacks privilege in _preview endpoint #72002 (issue: #72715)
  • Optmize histogam group_by change detection #74031 (issue: #63801)

Bug fixes

edit
Aggregations
  • Fix mapping error to indicate values field #74132
  • Prevent date_histogram from OOMing #72081 (issue: #71758)
  • Properly size empty filters #71864
  • Only enable "filter by filter" optimization for terms, date_histogram, range, and filters aggregation when we’re sure it’ll be faster #74260
CRUD
  • Move get-aliases handling onto management thread #74053
CompositeAggs
  • Fail composite aggregation if after key is unparsable #74252
Distributed
  • Change REST status code for TaskCancelledException to 400 #73524
Features/Ingest
  • Dissect parsing: An % occurring in the delimiter causes incorrect capture of the ${key} that follows #72876
  • Fix ignore_missing takes no effect in rename ingest processor #74248 (issue: #74241)
Features/Java High Level REST Client
  • Fix comparison of identical values leads to a bug of non-effective value #74040
Geo
  • Do not over-allocate when resizing in GeoHashTiler with bounds #72539 (issue: #72521)
  • Handle properly precision 0 for BoundedGeoTileGrid #72800
  • [Geopoint] Grid aggregations with bounds should exclude touching tiles #72493 (issue: #72295)
  • [Geoshape] Grid aggregations with bounds should exclude touching tiles #72295
  • GeoShapeIndexer#prepareForIndex should be called for computing the doc value centroid #73856 (issue: #73836)
Highlighting
  • Fix plain highlighter ordering for none #74084 (issue: #58236)
  • Fix highlighting for match_phrase_prefix query inside nested #73775 (issue: #70922)
Infra/Core
  • Change year max digits for strict_date_optional_time and date_optional_time #73034 (issues: #52396, #72191)
Infra/Logging
  • Fix deprecation logs throttling for deprecated routes #73051 (issue: #73002)
Infra/Scripting
  • Fix several function reference bugs in Painless #73538
Machine Learning
  • Make atomic operations safer for aarch64 #1893
  • Ensure bucket event_count is calculated for jobs with 1 second bucket spans #1909
Mapping
  • Check total field limit at parse time #73713 (issue: #73460)
Packaging
  • Allow container restarts with file logging #73101 (issue: #72702)
Recovery
Search
  • Dynamic runtime to not dynamically create objects #74234 (issue: #65489)
  • TermsEnum API - allow null search strings #73144 (issue: #73141)
Snapshot/Restore
  • Fix edge-case threading bug in TransportMountSearchableSnapshotAction #73196
  • Fix SnapshotInfo.fromXContentInternal not fully consuming parser #73268 (issue: #73149)
  • Fix SnapshotShardFailure.toXContent serialization issue #72801
TLS
  • Fix error reporting for SSL resources outside of config dir #64235 (issue: #63451)

Upgrades

edit
Features/Monitoring
Infra/Core
Network
  • Upgrade netty to 4.1.63 #73011
Packaging
  • Bump bundled JDK to 16.0.1 #73057