Elasticsearch version 7.13.0

edit

Elasticsearch version 7.13.0

edit

Also see Breaking changes in 7.13.

Known issues

edit
  • Frozen tier: (Windows only) The frozen data tier relies on multiple caching mechanisms to speed up access to searchable snapshot files. One of these caches uses sparse files to avoid creating large files on disk when it is not strictly required. A bug prevented files from being created with the right options to enable sparse support on Windows, leading Elasticsearch to create potentially large files that can end up consuming all the disk space.

    This issue is fixed in Elasticsearch versions 7.15.2 and later. There is no known workaround for earlier versions. Filesystems that enable sparse files by default are not affected. For more details, see #79371.

  • If autoscaling is enabled for machine learning, the administrator of the cluster should increase the cluster setting xpack.ml.max_open_jobs to the maximum value of 512. This allows autoscaling to run reliably as it relies on assigning jobs only via memory. Having xpack.ml.max_open_jobs as a small number may cause autoscaling to behave unexpectedly.
  • Snapshot and restore: If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail. To mitigate this problem, set snapshot.max_concurrent_operations: 1 to prevent concurrent snapshot operations:

    PUT _cluster/settings
    {
      "persistent" : {
        "snapshot.max_concurrent_operations" : 1
      }
    }

    This issue is fixed in Elasticsearch versions 7.13.1 and later. It is not possible to repair a repository once it is affected by this issue, so you must restore the repository from a backup, or clear the repository by executing DELETE _snapshot/<repository name>/*, or move to a fresh repository. For more details, see #73456.

  • If local and remote clusters are on different patch releases, response serialization fails for requests to a remote cluster that use the field capabilities API with CCS. Because Kibana uses this API internally, requests to CCS index patterns will also fail.

    This issue is fixed in Elasticsearch version 7.13.3 and later. For more details, see #74504.

Deprecations

edit
Aggregations
  • Deprecate date aggregations on boolean fields #70030 (issue: #59255)
Authentication
  • Deprecate the behavior of implicitly disabling file/native realm #69320 (issue: #50892)
Features/Indices APIs
  • Deprecate legacy index template API endpoints #71309 (issue: #71307)
Infra/Core
  • Add migration check for legacy role settings #71197 (issues: #54998, #71143)
  • Add multiple data paths deprecation to the deprecation API #71209 (issue: #71205)
  • Deprecate multiple path.data entries #71207 (issue: #71205)
Infra/Settings
  • Add clear deprecation around legacy role settings #71143 (issue: #54998)
  • Add deprecation warning for default value of action.destructive_requires_name #70932 (issues: #66908, #67543)

New features

edit
EQL
  • Adds fields request field to the EQL request #68962 (issue: #68115)
Features/ILM+SLM
  • Make all the shrink action steps retryable #70107 (issue: #48183)
Features/Ingest
  • Add support for validating IPv4/IPv6 addresses to Convert processor #69989 (issue: #36145)
  • Add registered domain processor #67611
Infra/Core
  • Add API for resetting state of a SystemIndexPlugin #69469
Machine Learning
  • Adds new trained model alias API to simplify trained model updates and deployments #68922
Search

Enhancements

edit
Aggregations
  • Improve handling of incompatible after key types in composite aggregations #70839 (issue: #70480)
  • Increase search.max_bucket default value by one #70645 (issue: #57042)
  • Modest memory savings in a date_histogram followed by a terms aggregation #68592
  • Speed up top_metrics on hot shards #70579 (issue: #70453)
  • Speed up aggregations with sub-aggregations #69806 (issues: #63643, #68871)
  • Speed up terms aggregation when alone #69377 (issue: #68871)
  • Speed up terms aggregation when not force merged #71241 (issue: #71086)
  • Use #updateTop to speed up InternalComposite#reduce #71278
Allocation
  • Improve awareness allocation explanation #69371
  • Skip zone/host awareness with auto-expand replicas #69334 (issues: #2869, #54151)
Analysis
  • Allow updateable flag for keyword_marker filter #65457 (issue: #65355)
Audit
Authentication
  • Add a deprecation message if a REST wrapper implementing plugin presents #66827
  • Service Accounts - Authentication with file tokens #70543
  • Service Accounts - Fleet integration #70724
  • Service Accounts - Get service account API #71315
  • Service Accounts - Initial bootstrap plumbing to add essential classes #70391
  • Service Accounts - New CLI tool for managing file tokens #70454
  • Service Accounts - delete index backed service account token #71165
  • Service Accounts - token name in response to Authenticate API #71382
  • Support metadata on API keys #70292 (issue: #48182)
Authorization
  • Add read permissions for apm_user role to APM fleet indices #68749
  • Include role names in access denied errors #69318 (issue: #42166)
Autoscaling
  • Frozen tier autoscaling decider based on shards #71042
Cluster Coordination
  • Include node roles in cluster state JSON response #71386 (issue: #71385)
  • Remove node attributes from cluster membership messages #69811
Distributed
  • Add fleet polling API for global checkpoint #71093
  • Reduce size of MANAGEMENT threadpool on small node #71171 (issue: #70435)
EQL
  • Allow Unicode escape sequences in strings #70514 (issue: #62832)
  • Improve null handling in the optimizer #70557
Engine
  • Support include_unloaded_segments in node stats #69682
Features/ILM+SLM
  • Add max_single_primary_size as a condition for the ILM rollover action #68917 (issues: #63026, #67842)
  • Reject creating ILM policies with phase timings that are not greater than or equal to the previous phase #70089 (issue: #70032)
  • Switch built-in policies to max_primary_shard_size #69995 (issue: #63026)
Features/Indices APIs
  • Date math support for aliases #67226 (issue: #20367)
  • Introduce separate shard limit for frozen shards #71392 (issues: #34021, #71042)
  • Support specifying multiple templates names in delete component template api #70314 (issue: #69973)
  • Support specifying multiple templates names in delete composable index template api #70094 (issue: #69973)
Features/Ingest
  • Accept more ingest simulate params as integers or strings #66197 (issues: #23823, #65992)
  • Extract device type from user agent info #69322
  • Network direction processor additions #68712
  • Summary option for listing ingest pipelines without their definitions #69756 (issue: #31954)
  • MurmurHash3 support for fingerprint processor #70632 (issue: #69182)
Features/Java Low Level REST Client
  • Support new data roles #66947
Features/Stats
Features/Watcher
  • Migrate watcher to system indices infrastructure #67588 (issue: #61656)
Infra/Core
  • Manage Fleet system indices within Elasticsearch #70689
  • Support mixed node versions in system index descriptors #71144
Infra/Logging
  • Only install templates for deprecation indices from elected master node #70057 (issues: #69918, #70020)
Infra/Scripting
  • Add Runtime Fields Contexts to Painless Execute API #71374 (issue: #70467)
  • Add a new ANTLR lexer for Painless suggestions #70517
  • Improve null def access error message #69226 (issue: #53129)
  • Make the available ScriptContexts accessible from ScriptService #70465
  • Whitelist the CIDR convenience API #71258 (issue: #60668)
Query Languages
  • Adds runtime_mappings to EQL and SQL requests #71356 (issue: #68116)
Machine Learning
  • Add new delete trained model aliases API #69195
  • Add runtime mappings to data frame analytics source config #69183 (issue: #65056)
  • Adding new _preview endpoint for data frame analytics #69453
  • Adding support for composite aggregations in anomaly detection #69970
  • Allow datafeed and job configs for datafeed preview API #70836 (issue: #70264)
  • Improve messages related to assigning machine learning jobs #69752
  • Add put and delete trained model alias APIs to high-level REST client #69214
  • Speed up training of regression and classification models for data sets with many features #1746
  • Adjust the syscall filter to allow mremap and avoid spurious audit logging #1819
  • Avoid overfitting in final training by scaling regularizers to account for the difference in the number of training examples. This results in a better match between train and test errors for classification and regression and often slightly improved test errors #1755
Mapping
  • Add calculated numeric fields #69531
  • Allow specify dynamic templates in bulk request #69948 (issue: #61939)
  • Change default format for date_nanos field #70463 (issues: #67063, #69192)
  • Expose if a field is a metadata field in the field capabilities response #69977
  • Field capabilities index action should not fork its execution #69865
  • Improve error message for invalid field name #70972 (issue: #70960)
  • New queryable _tier metadata field #69288 (issue: #68135)
  • Output script stats for indexed fields #71219
  • Preserve half_float precision in fields API #70653 (issue: #70260)
  • Support fetching flattened subfields #70916 (issue: #70605)
Network
Ranking
  • Add access to dense_vector values #71313 (issue: #51964)
  • Make wildcard field use constant scoring queries for wildcard queries #70452 (issue: #69604)
Recovery
  • Fix retention lease expiry to not mark stale #68577
SQL
  • Removed the always on total hit tracking #70319 (issue: #52787)
Search
  • Add _size and _doc_count to fields output #70575 (issue: #63569)
  • Add positive_score_impact to rank_features type #69994 (issue: #68619)
  • Add earlier validation for some SearchSourceBuilder settings #69548 (issue: #54958)
  • Allow format sort values of date fields #70357 (issue: #69192)
  • Cancel searches earlier #69795
  • Close search contexts on reassigned shard #68539
  • Improve lookup for include_unmapped field pattern #69984 (issue: #69983)
  • Support fetching _tier field value #71379 (issues: #63569, #68135)
Security
  • Service Accounts - CLI to delete and list file tokens #71380
  • Warn users if security is implicitly disabled #70114
Snapshot/Restore
  • Add searchable snapshot stats for reads from Lucene #70464
  • Add support for range reads and retries to URL repositories #69521
  • Change tier preference for shared_cache searchable snapshots to frozen only #70786 (issue: #70341)
  • Enforce data_frozen for partial searchable snapshot _tier_preference #71155 (issues: #70786, #71014)
  • Forbid dedicated frozen nodes w/ unfrozen indices #71395
  • Include min/max/average file size in Searchable Snapshots Stats API #70294
  • Make searchable snapshot cache size effectively zero on non-frozen nodes #71134 (issues: #70341, #70846, #71013)
  • Skip TRANSLOG stage for searchable snapshots recovery stage #70311 (issue: #65531)
  • Use default application credentials for GCS repositories #71239
Transform
  • Add support for geo_line aggregation in pivot function #69299
  • Enhance transform role checks #70139 (issue: #69518)
  • Redirect transform actions to transform and remote_cluster_client node when needed #70125
  • Report warnings in _preview response #68396 (issue: #70059)

Bug fixes

edit
Aggregations
  • Significant text aggregation - return empty results rather than error if field unmapped #70778 (issue: #69809)
  • Stop terms aggregation from losing buckets #70493 (issues: #68871, #70449)
Analysis
  • Ukrainian language plugin can fill up heap #71998
Authentication
  • Fix inconsistency of internal user checking #70123
Engine
  • Allow force-merges to run in parallel on a node #69416 (issue: #69327)
Features/Features
  • Unique names for bulk processor scheduler threads #69432 (issues: #1, #68470)
Features/Java High Level REST Client
  • Fix ignoring require_alias parameter in high level rest client #67865 (issue: #67819)
Features/Java Low Level REST Client
  • Fix Suppressing Interrupted Flag in Client #68999 (issue: #68525)
Geo
  • Fix overflow in GeoTileGridTiler #70222
Infra/Logging
  • Add RateLimitingFiltering to plaintext deprecation logs #69190 (issues: #61474, #69188)
  • Do not throttle deprecated field logs #70009 (issue: #55115)
Infra/Scripting
Machine Learning
  • Consider xpack.ml.max_ml_node_size in effective_model_memory_limit #70473 (issue: #70069)
  • Do not create machine learning annotations index in upgrade mode #71175
  • Do not track machine learning usage when collecting monitoring #71314
  • Ensure auc_roc curve is monotonic #70628
  • Exclude nested fields in data frame analytics #71400
  • Improve readability of messages written when assigning machine learning jobs to nodes #69629 (issue: #59602)
  • Ensure the same hyperparameters are chosen if classification or regression training is stopped and restarted, for example, if the node fails #1848
  • Fail gracefully if insufficient data is supplied for classification or regression training #1855
  • Fail gracefully on encountering unexpected state in restore from snapshot for anomaly detection #1872
  • Use appropriate memory ordering flags for aarch64 with string store to avoid excessive string duplication #1888
  • Fix autoscaling bug where many jobs take a long time to open #72423
  • Use appropriate master timeouts for master actions #72492
  • Fix empty overall_buckets response #72542
  • Check the out stream exists before consuming it #72455
  • Prevent data frame analytics freeze after loading data #72412
Mapping
  • Fix binary docvalue_fields with padding #70826 (issue: #70244)
  • Propagate index errors in field_caps #70245 (issue: #68994)
Search
  • Correct service time parameter in ARS formula #70283 (issue: #65838)
  • In ARS, correct default number of outstanding requests #71022 (issue: #70283)
  • Prevent aliased fields being used for index sorts #70879
Transforms
  • Fix bug where group_by ordering could break when serializing between nodes #72016
  • Avoid transform failure during rolling upgrade #72533