IMPORTANT: No additional bug fixes or documentation updates
will be released for this version. For the latest information, see the
current release documentation.
Elasticsearch version 7.16.0
editElasticsearch version 7.16.0
editAlso see Breaking changes in 7.16.
Deprecations
edit- Allocation
- Authentication
- Cluster Coordination
- Geo
- ILM+SLM
- Indices APIs
- Infra/Logging
-
-
Change default deprecation logger level to
CRITICAL
#77030 (issue: #76754) - Deprecation logs indexing is enabled by default #78991 (issues: #76292, #78850)
- Exposing the ability to log deprecated settings at non-critical level #79107 (issues: #77482, #78781)
- REST API to reset deprecation indexing cache #78392 (issue: #78134)
- Add category field to deprecation logs #77550 (issue: #67443)
-
Change default deprecation logger level to
- Infra/Scripting
- Java High Level REST Client
-
- Add deprecation headers to HLRC classes #79754
- Machine Learning
-
-
Deprecate
estimated_heap_memory_usage_bytes
and replace withmodel_size_bytes
#80545
-
Deprecate
- Monitoring
Known issues
edit-
Painless: Any Painless script where a method call resolves a lambda argument to a
UnaryOperator
or aBinaryOperator
type results in anIllegalArgumentException
. This exception results in a compilation error when static types are used, or a runtime error for dynamic types. Use standard procedural statements as a workaround, such asif/else
,while
, andfor
statements.The following example is a
UnaryOperator
type that will result in a failure:List l = ['abc', '123']; l.replaceAll(f -> f.replace('abc', 'doremi');
The following example is a possible workaround:
List l = ['abc', '123']; for (int i = 0; i < l.size(); ++i) { l.set(i, l.get(i).replace('abc', 'doremi')); }
New features
editEnhancements
edit- Aggregations
- Allocation
- Audit
- Authentication
-
- Remove HTTPS check for API Keys & Service Accounts #76801
- Authorization
-
-
Add privileges to
kibana_system
for upgrading endpoint hidden indices #80140 - Allow fleet-server service account to set up Fleet #78192 (issue: #78078)
-
Grant additional privileges for endpoint transform indices to
kibana_system
#79619 -
Grant privileges required by package upgrade to
kibana_system
#78049 (issue: #77294) -
Optimize FLS/DLS setup in
IndicePermission
authz #77832 - Skip loading authorized indices if requests do not need them #78321
-
Superuser fastpath for
indexAccessControl
#78498 -
Update transform destination index privilege for
kibana_system
#79076 -
Use fixed size memory allocation in
IndicesPermission
#77748 - Add proper permissions to fleet server for Endpoint response index #80231
- Migrate custom role providers to licensed feature #79127
-
Add privileges to
- CCR
- Cluster Coordination
-
- Add cluster state serialization stats #78816
- Add timing stats to publication process #76771 (issue: #76625)
- Get hot threads on lagging nodes #78879
- Improve error message in 8.x to 7.x downgrade #78644 (issue: #78638)
-
Improve logging in
LeaderChecker
#78883 - Mention "warn threshold" in master service slowlog #76815 (issue: #76625)
- Recycle pages used by outgoing publications #77407 (issue: #77317)
- Reuse previous indices lookup when possible #79004 (issues: #77888, #78980)
- Validate PING and STATE connections at join time #77741
- Composite aggregations
- Data streams
- Distributed
- EQL
- Engine
- Geo
- ILM+SLM
-
-
Improve
LifecycleExecutionState
parsing #77855 (issue: #77466) -
Reduce the number of times that
LifecycleExecutionState
is parsed when running a policy #77863 (issue: #77466) - Add built-in ILM policies for common user use cases #76791
- Allow for setting the total shards per node in the Allocate ILM action #76134
-
Improve
- Indices APIs
- Infra/Core
-
-
Add optional content checking to
ResourceWatcher
#79423 -
Add other time accounting in
HotThreads
#79392 -
Add upgrade preparation and completion callbacks to
SystemIndexPlugin
#78542 -
Cache
index.hidden
setting #78612 (issue: #77974) - Enable wait/blocked time accounting #77935 (issue: #72376)
- Handle empty /proc/self/cgroup file #78659 (issue: #77833)
- Implement framework for migrating system indices #78951
- Require System Index Descriptors to allow a specific suffix #78355
-
Support mem type in nodes
hot_threads
API #72850 (issue: #70345) -
Use enum field for
HotThreads
report type #77462
-
Add optional content checking to
- Infra/Node Lifecycle
-
- Enable exit on out of memory error #71542
- Infra/Scripting
-
- Add a direct sub classes data structure to the Painless lookup #76955
- Add ability to augment classes with fields from other classes in Painless #76628
- Add dynamic (duck) type resolution to Painless static types #78575
- Adds a lookup method to Painless for finding methods of all sub classes #77044
-
UnsignedLong
field type converter #77271 - Compile/cache eviction history metric placeholders #78257 (issue: #62899)
- Infra/Settings
- Ingest
-
- Add enrich node cache #76800 (issue: #48988)
- Add indices pipeline settings check when deleting a pipeline #77013
- Allow range types to be used for enrich matching #76110
- ECS support for Grok processor #76885 (issue: #66528)
- Improving cache lookup to reduce recomputing / searches #77259
- Make enrich policy execution cancelable #77188 (issue: #48988)
- Optimistic concurrency control for updating ingest pipelines #78551 (issue: #77031)
- Sync grok processor patterns with Logstash #76752
- Updating ingest pipeline without changes is no-op #78196 (issue: #77382)
- Java High Level REST Client
- License
- Machine Learning
-
-
Add new
normalize_above
parameter top_value
significant terms heuristic #78833 -
Add new default char filter
first_line_with_letters
for machine learning categorization #77457 -
Add new
defer_definition_decompression
parameter to put trained model API #77189 (issue: #77132) - Enable ML on macOS on ARM #78203
- Track feature usage for data frame analytics, inference, and anomaly jobs #76789
- Speed up training of regression and classification models #2024
- Improve concurrency for training regression and classification models #2031
-
Improve aspects of implementation of
skip_model_update
rule #2053 - Make sure instrumentation captures the best hyperparameters found for training classification and regression models https://github.com/elastic/ml-cpp/pull/2057{#2057}
-
Add new
- Mapping
- Monitoring
-
- Remove license check for monitoring data retention #79010
- Packaging
- Recovery
- Search
-
-
Add
_ignored
and_routing
metatada fields to fields API #78981 (issues: #75836, #78828) -
Add
_index
and_version
metatada fields to fields API #79042 (issues: #75836, #78828) -
Add ability to retrieve
_id
via fields option #78828 (issue: #75836) - Add node-level field capabilities API requests #79212 (issues: #74648, #77047, #78647)
- Add segment sorter for data streams #75195
- Add sort optimization with after from Lucene #64292
- Don’t always rewrite the Lucene query in search phases #79358
- Expand segment sorter for all timeseries indices #78639 (issue: #75195)
- Node level can match action #78765
- Search - return ignored field values from fields API #78697 (issue: #74121)
- Support request cache on frozen tier #77694 (issue: #75309)
-
Use
search_coordination
threadpool for field capabilities API requests #79378 (issue: #79212) - Create a sha-256 hash of the shard request cache key #74877 (issue: #74061)
-
Add
- Security
- Snapshot/Restore
-
- Add filtering by SLM policy to get snapshots API #77321
- Add sort by shard count and failed shard count to get snapshots API #77011
- Add descriptions to various tasks #76700
- Add maintenance service to clean up unused docs in snapshot blob cache #77686
- Add periodic maintenance task to clean up unused blob store cache docs #78438 (issue: #77686)
-
Filter Unneeded
SnapshotInfo
Instances Early inTransportGetSnapshotsAction
#78032 (issue: #74350) - Implement exclude patterns for snapshot and repository names in get snapshots API #77308
- Implement sort by repository name in get snapshots API #77049
-
Implement
from_sort_value
parameter in get snapshots API #77618
- Stats
- Transform
Bug fixes
edit- Aggregations
-
- Add extra round trip to aggregation tests #79638 (issue: #73680)
-
Fix rate aggregation with custom
_doc_count
#79346 (issue: #77734) - Fix several potential circuit breaker leaks in aggregators #79676
- Scale doubles to floats when necessary to match the field #78344 (issue: #77033)
- Support for aggregation names with dots in first element path of a pipeline aggregation #77481
- Allocation
- Authorization
- CAT APIs
-
-
Adjust
_cat/templates
to not request all metadata #78829
-
Adjust
- CCR
- CRUD
- Cluster Coordination
-
- Avoid early release of local forking requests #77641 (issues: #77407, #77634)
-
Check for global blocks after
IndexNotFoundException
inTransportMasterNodeAction
#78128 (issue: #70572) - Improve control of outgoing connection lifecycles #77295 (issue: #67873)
- Only remove active peer on connection failure #79557 (issues: #77295, #79550)
-
Reduce merging in
PersistedClusterStateService
#79793 (issue: #77466)
- CompositeAggs
- Data streams
-
- Add replicated field to get data stream API response #80988 (issue: #118899)
- Correct check for write index and increment generation on all data stream backing index operations #79916
- Fix data stream bug causing it to rollover to a non-existent Index #79759
-
Fix
IndexNotFoundException
error when handling remove alias action #80312 - Fix data stream alias validation #81040 (issue: #80972)
- Distributed
- EQL
-
- Add optional fields and limit joining keys on non-null values only #79677
- Geo
- ILM+SLM
- Indices APIs
- Infra/Core
-
- Avoid spurious deprecation warnings when calling Deprecation Info API #78151 (issue: #78098)
-
Fix overflow/underflow in
CompositeBytesReference
#78893 - Fix race condition in Feature Migration Status API #80572 (issue: #79680)
- Prevent stack overflow in rounding #80450
-
Set
LIBFFI_TMPDIR
at startup #80651 (issues: #18272, #73309, #74545, #77014, #77053, #77285, #80617) - Strip index blocks from settings for reindex targets #80887 (issue: #80654)
- Eschew leniency when parsing time zones #77267 (issues: #73955, #76415)
- Fix ingest timezone parsing #63876 (issue: #63458)
- Infra/Logging
- Infra/Settings
- Ingest
- Java High Level REST Client
- Machine Learning
-
- Address potential ML feature reset permissions bug #79179
- Adjust ML memory tracker to reduce logging impact #78482
- Audit job open failures and stop any corresponding datafeeds #80665 (issues: #48934, #80621)
- Fix acceptable model snapshot versions in ML deprecation checker #81060 (issues: #79387, #81039, #119745)
- Fix autoscaling capacity consistency #81181
- Fix bug in inference stats persister for when feature reset is called #77213 (issue: #77182)
- Fix datafeed preview with remote indices #81099 (issue: #77109)
- Fix language identification bug when multi-languages are present #80675
-
Fix model snapshot sorting when sorting by
min_version
#80596 (issue: #80561) - ML legacy index templates that are no longer needed should be deleted #80874 (issue: #80876)
- Need to tolerate .ml-config being an alias #80025
- Parent datafeed actions to the datafeed’s persistent task #81143
- Wait for .ml-state-write alias to be readable #79731 (issue: #79636)
- Improve the estimates of hyperparameter importance in data frame analytics and avoid incorrectly stopping the hyperparameter search prematurely #2073
- Fix numerical instability in hyperparameter optimization for training regression and classification models #2078
- Fix numerical stability issues in time series modeling #[2083]
- Mapping
- Packaging
-
-
Ensure
LIBFFI_TMPDIR
is exported by init script #80794
-
Ensure
- Recovery
- SQL
- Search
-
- Minimize search source of shard level search requests #80634 (issue: #80187)
-
Prevent
NullPointerException
inSourceConfirmedTextQuery
#80472 (issue: #80419) -
_terms_enum
APIindex_filter
doesn’t work with_tier
field on upgraded cluster #79553 (issue: #79200) - Undeprecate the auto complete thread pool #80204
- Wildcard field regex query fix #78839 (issue: #78391)
- Snapshot/Restore
-
- Fix Queued Snapshot Clone not Starting after Data Node Drops Out #77111 (issue: #77101)
- Fix Temporarily Leaking Shard Level Metadata Blobs in some Cases #76562
- Fix after restore Lucene.pruneUnreferencedFiles() conditional #81047 (issues: #68821, #75308)
-
Improve handling of corrupt
index.latest
blob #77339 - Submit GCS delete batch requests incrementally #80540
- Fix repository-azure for empty settings on reload #79559
- Transform
- Watcher