A newer version is available. For the latest information, see the
current release documentation.
Elasticsearch version 7.6.0
editElasticsearch version 7.6.0
editAlso see Breaking changes in 7.6.
Known issues
edit- Applying deletes or updates on an index after it has been shrunk may corrupt the index. In order to prevent this issue, it is recommended to stop shrinking read-write indices. For read-only indices, it is recommended to force-merge indices after shrinking, which significantly reduces the likeliness of this corruption in the case that deletes/updates would be applied by mistake. This bug is fixed in Elasticsearch 7.7 and later versions. More details can be found on the corresponding issue.
-
Indices created in 6.x with
date
anddate_nanos
fields using formats that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. https://github.com/elastic/elasticsearch/pull/52555 This is fixed in Elasticsearch 7.7 and later versions. -
Slow loggers can cause Log4j loggers to leak over time. When a new index is created, a new Log4j logger is associated with it. However, when an index is deleted, Log4j keeps an internal reference to its loggers that results in a memory leak (issue: #56171)
This issue is fixed in Elasticsearch 6.8.10 and 7.7.1.
-
Week-based date patterns are not working correctly with
Y
. UsingY
withw
will result in a failed request and an exception in the logs (issue: #57128). Usingy
withw
results in incorrect date calculations. A workaround is to add the following line to thejvm.options
file.9-:-Djava.locale.providers=SPI,COMPAT
This issue is fixed in Elasticsearch 7.7.0 and later versions (issue: #50916).
Breaking changes
editBreaking Java changes
editDeprecations
edit- Analysis
- Authorization
-
- Deprecating kibana_user and kibana_dashboard_only_user roles #46456
- Distributed
- Features/Indices APIs
- Infra/Core
- Reindex
- Search
New features
edit- Aggregations
- Analysis
- Authentication
-
- Password Protected Keystore (Feature Branch) #49210
- Features/ILM+SLM
- Features/Ingest
- Machine Learning
-
-
Implement
precision
andrecall
metrics for classification evaluation #49671 (issue: #48759) - Explain data frame analytics API #49455
- Machine learning model inference ingest processor #49052
- Implement accuracy metric for multi-class classification #47772 (issue: #48759)
- Add feature importance values to classification and regression results (using tree SHapley Additive exPlanation, or SHAP) #857
-
Implement
- Mapping
- Search
Enhancements
edit- Aggregations
- Allocation
- Analysis
- Authentication
- Authorization
-
- Increase Size and lower TTL on DLS BitSet Cache #50535 (issues: #43669, #49260)
- Add monitor_snapshot cluster privilege #50489 (issue: #50210)
- Remove reserved roles for code search #50068 (issue: #49842)
- [Code] Remove code_admin/code_user roles #48164
- Resolve the role query and the number of docs lazily #48036
- CCR
- CRUD
- Cluster Coordination
- Distributed
- Engine
-
- Do not force refresh when write indexing buffer #50769
- Deleted docs disregarded for if_seq_no check #50526
- Allow realtime get to read from translog #48843
- Do not warm up searcher in engine constructor #48605 (issue: #47186)
- Add a new merge policy that interleaves old and new segments on force merge #48533 (issue: #37043)
- Refresh should not acquire readLock #48414 (issue: #47186)
- Features/ILM+SLM
-
- Refresh cached phase policy definition if possible on new poli… #50820 (issue: #48431)
- Make the UpdateRolloverLifecycleDateStep retryable #50702 (issue: #48183)
- Make InitializePolicyContextStep retryable #50685 (issue: #48183)
- ILM retryable async action steps #50522 (issues: #44135, #48183)
- Make the TransportRolloverAction execute in one cluster state update #50388
- ILM open/close steps are noop if idx is open/close #48614
-
ILM Make the
check-rollover-ready
step retryable #48256 (issue: #44135)
- Features/Ingest
-
- Foreach processor - fork recursive call #50514
- Sync grok patterns with logstash patterns #50381
- Replace required pipeline with final pipeline #49470 (issue: #49247)
- Add templating support to enrich processor #49093
- Introduce on_failure_pipeline ingest metadata inside on_failure block #49076 (issue: #44920)
- Add templating support to pipeline processor. #49030 (issue: #39955)
- Add option to split processor for preserving trailing empty fields #48664 (issue: #48498)
- Change grok watch dog to be Matcher based instead of thread based. #48346 (issues: #43673, #47374)
- update ingest-user-agent regexes.yml #47807
- Features/Java High Level REST Client
- Features/Monitoring
-
- Significantly Lower Monitoring HttpExport Memory Footprint #48854
- Validate proxy base path at parse time #47912 (issue: #47711)
- Validate index name time format setting at parse time #47911 (issue: #47711)
- Validate monitoring header overrides at parse time #47848 (issue: #47711)
- Validate monitoring username at parse time #47821 (issue: #47711)
- Validate monitoring password at parse time #47740 (issue: #47711)
- Features/Stats
- Features/Watcher
- Geo
- Infra/Core
- Infra/Packaging
-
- Respect ES_PATH_CONF on package install #50158
- Restrict support for CMS to pre-JDK 14 #49123 (issue: #46973)
- Remove parsed JVM settings from general settings in Windows service daemon manager #49061 (issue: #48796)
- Package the JDK into jdk.app on macOS #48765
- Add UBI-based Docker images #48710 (issue: #48429)
- Infra/Plugins
- Infra/Scripting
-
- Scripting: ScriptFactory not required by compile #50344 (issue: #49466)
- Scripting: Cache script results if deterministic #50106 (issue: #49466)
- Scripting: Groundwork for caching script results #49895 (issue: #49466)
- Scripting: add available languages & contexts API #49652 (issue: #49463)
- Scripting: fill in get contexts REST API #48319 (issue: #47411)
- Scripting: get context names REST API #48026 (issue: #47411)
- Infra/Settings
- License
- Machine Learning
-
- Add audit warning for 1000 categories found early in job #51146 (issue: #50749)
-
Add
num_top_feature_importance_values
param to regression and classification #50914 - Implement force deleting a data frame analytics job #50553 (issue: #48124)
- Delete unused data frame analytics state #50243
- Make each analysis report desired field mappings to be copied #50219 (issue: #50119)
- Retry bulk indexing of state docs #50149 (issue: #50143)
- Persist/restore state for data frame analytics classification #50040
-
Introduce
randomize_seed
setting for regression and classification #49990 -
Pass
prediction_field_type
to C++ analytics process #49861 (issue: #49796) - Add optional source filtering during data frame reindexing #49690 (issue: #49531)
- Add default categorization analyzer definition to ML info #49545
- Add graceful retry for anomaly detector result indexing failures #49508 (issue: #45711)
- Lower minimum model memory limit value for data frame analytics jobs from 1MB to 1kB #49227 (issue: #49168)
-
Improve
model_memory_limit
user experience for data frame analytics jobs #44699 - Improve performance of boosted tree training for both classification and regression #775
- Reduce the peak memory used by boosted tree training and fix an overcounting bug estimating maximum memory usage #781
- Stratified fractional cross validation for regression #784
-
Added
geo_point
supported output forlat_long
function records #809, #47050 - Use a random bag of the data to compute the loss function derivatives for each new tree which is trained for both regression and classification #811
-
Emit
prediction_probability
field alongside prediction field in ml results #818 - Reduce memory usage of machine learning native processes on Windows #844
- Reduce runtime of classification and regression #863
- Stop early training a classification and regression forest when the validation error is no longer decreasing #875
-
Emit
prediction_field_name
in data frame analytics results using the type provided asprediction_field_type
parameter #877 - Improve performance updating quantile estimates #881
- Migrate to use Bayesian optimisation for initial hyperparameter value line searches and stop early if the expected improvement is too small #903
- Stop cross-validation early if the predicted test loss has a small chance of being smaller than for the best parameter values found so far #915
- Optimize decision threshold for classification to maximize minimum class recall #926
-
Include categorization memory usage in the
model_bytes
field inmodel_size_stats
, so that it is taken into account in node assignment decisions #927 (issue: #724)
- Mapping
-
- Add telemetry for flattened fields. #48972
- Network
-
- Add certutil http command #49827
- Do not load SSLService in plugin contructor #49667 (issue: #44536)
- Netty4: switch to composite cumulator #49478
- Add the simple strategy to cluster settings #49414 (issue: #49067)
- Deprecate misconfigured SSL server config #49280 (issue: #45892)
- Improved diagnostics for TLS trust failures #48911
- Percolator
- Ranking
- Recovery
- Reindex
- SQL
-
- SQL: Handle uberjar scenario where the ES jdbc driver file is bundled in another jar #51856 (issue: #50201)
- SQL: add trace logging for search responses coming from server #50530
- SQL: Add TRUNC alias for TRUNCATE #49571 (issue: #41195)
- SQL: binary communication implementation for drivers and the CLI #48261 (issue: #47785)
- SQL: Verify Full-Text Search functions not allowed in SELECT #51568 (issue: #47446)
- Search
-
- Add Validation for maxQueryTerms to be greater than 0 for MoreLikeThisQuery #49966 (issue: #49927)
- Optimize numeric sort on match_all queries #49717 (issue: #48804)
- Pre-sort shards based on the max/min value of the primary sort field #49092 (issue: #49091)
- Optimize sort on long field #48804
- Search optimisation - add canMatch early aborts for queries on "_index" field #48681 (issue: #48473)
- #48475 Pure disjunctions should rewrite to a MatchNoneQueryBuilder #48557
- Disable caching when queries are profiled #48195 (issue: #33298)
- BlendedTermQuery’s equals method should consider boosts #48193 (issue: #48184)
- Increase the number of vector dims to 2048 #46895
- Security
- Snapshot/Restore
-
- Use Cluster State to Track Repository Generation #49729 (issue: #49060)
- Track Repository Gen. in BlobStoreRepository #48944 (issues: #38941, #47520, #47834, #49048)
- Restore from Individual Shard Snapshot Files in Parallel #48110 (issue: #42791)
- Track Shard-Snapshot Index Generation at Repository Root #46250 (issues: #38941, #45736)
- Store
- Transform
Bug fixes
edit- Aggregations
-
- Use #name() instead of #simpleName() when generating doc values #51920 (issues: #50307, #51847)
- Fix a sneaky bug in rare_terms #51868 (issue: #51020)
- Support time_zone on composite’s date_histogram #51172 (issues: #45199, #45200)
- Fix format problem in composite of unmapped #50869 (issue: #50600)
- SingleBucket aggs need to reduce their bucket’s pipelines first #50103 (issue: #50054)
- Avoid precision loss in DocValueFormat.RAW#parseLong #49063 (issue: #38692)
- Fix ignoring missing values in min/max aggregations #48970 (issue: #48905)
- Allocation
- Analysis
- Audit
- Authentication
-
- Preserve ApiKey credentials for async verification #51244
- Don’t fallback to anonymous for tokens/apikeys #51042 (issue: #50171)
- Populate User metadata with OpenIDConnect collections #50521 (issue: #50250)
- Always return 401 for not valid tokens #49736 (issue: #38866)
- Fix iterate-from-1 bug in smart realm order #49473
- Remove unnecessary details logged for OIDC #48746
- Add owner flag parameter to the rest spec #48500 (issue: #48499)
- Authorization
- CCR
-
- CCR should auto-retry rejected execution exceptions #49213
- CRUD
-
- Block too many concurrent mapping updates #51038 (issue: #50670)
- Ensure meta and document field maps are never null in GetResult #50112 (issue: #48215)
- Replicate write actions before fsyncing them #49746
- Do not mutate request on scripted upsert #49578 (issue: #48670)
- Fix Transport Stopped Exception #48930 (issue: #42612)
- Return consistent source in updates #48707
- Close query cache on index service creation failure #48230 (issue: #48186)
- Cluster Coordination
- Discovery-Plugins
- Distributed
- Engine
-
- Do not wrap soft-deletes reader for segment stats #51331 (issues: #51192, #51303)
- Account soft-deletes in FrozenEngine #51192 (issue: #50775)
- Account trimAboveSeqNo in committed translog generation #50205 (issue: #49970)
- Greedily advance safe commit on new global checkpoint #48559 (issue: #48532)
- Do not ignore exception when trim unreferenced readers #48470
- Features/Features
-
- Fix X-Pack SchedulerEngine Shutdown #48951
- Features/ILM+SLM
-
- Fix SLM check for restore in progress #50868
- Handle failure to retrieve ILM policy step better #49193 (issue: #49128)
- Don’t halt policy execution on policy trigger exception #49128
- Re-read policy phase JSON when using ILM’s move-to-step API #48827
- Don’t schedule SLM jobs when services have been stopped #48658 (issue: #47749)
- Ensure SLM stats does not block an in-place upgrade from 7.4 #48367
- Ensure SLM stats does not block an in-place upgrade from 7.4 #48361
- Add SLM support to xpack usage and info APIs #48096 (issue: #43663)
- Change policy_id to list type in slm.get_lifecycle #47766 (issue: #47765)
- Features/Ingest
-
- Fix ignore_missing in CsvProcessor #51600
- Don’t overwrite target field with SetSecurityUserProcessor #51454 (issue: #51428)
- Fix ingest simulate response document order if processor executes async #50244
- Allow list of IPs in geoip ingest processor #49573 (issue: #46193)
- Do not wrap ingest processor exception with IAE #48816 (issue: #48810)
- Introduce dedicated ingest processor exception #48810 (issue: #48803)
- Features/Java High Level REST Client
- Features/Java Low Level REST Client
- Features/Monitoring
- Geo
-
- Guard against null geoBoundingBox #50506 (issue: #50505)
- Geo: Switch generated GeoJson type names to camel case (#50285) #50400 (issue: #49568)
- Geo: Switch generated WKT to upper case #50285 (issue: #49568)
- Fix typo when assigning null_value in GeoPointFieldMapper #49645
- Fix handling of circles in legacy geo_shape queries #49410 (issue: #49296)
- GEO: intersects search for geo_shape return wrong result #49017
- Geo: improve handling of out of bounds points in linestrings #47939 (issue: #43916)
- Highlighting
-
- Fix invalid break iterator highlighting on keyword field #49566
- Infra/Core
-
- Ignore virtual ethernet devices that disappear #51581 (issue: #49914)
- Guess root cause support unwrap #50525 (issue: #50417)
- Allow parsing timezone without fully provided time #50178 (issue: #49351)
- [Java.time] Retain prefixed date pattern in formatter #48703 (issue: #48698)
- Don’t drop user’s MaxDirectMemorySize flag on jdk8/windows #48657 (issues: #44174, #48365)
- Warn when MaxDirectMemorySize may be incorrect (Windows/JDK8 only issue) #48365 (issue: #47384)
- [Java.time] Calculate week of a year with ISO rules #48209 (issues: #41670, #42588, #43275, #43652)
- Infra/Logging
- Infra/Packaging
- Infra/REST API
- Machine Learning
-
- Fix 2 digit year regex in find_file_structure #51469
-
Validate classification
dependent_variable
cardinality is at least two #51232 - Do not copy mapping from dependent variable to prediction field in regression analysis #51227
- Handle nested and aliased fields correctly when copying mapping #50918 (issue: #50787)
-
Fix off-by-one error in
ml_classic
tokenizer end offset #50655 - Improve uniqueness of result document IDs #50644 (issue: #50613)
- Fix accuracy metric in multi-class confusion matrix #50310 (issue: #48759)
- Fix race condition when stopping a data frame analytics jobs immediately after starting it #50276 (issues: #49680, #50177)
- Apply source query on data frame analytics memory estimation #49517 (issue: #49454)
- Fix r_squared eval when variance is 0 #49439
- Blacklist a number of prediction field names #49371 (issue: #48808)
- Make data frame analytics more robust for very short-lived analyses #49282 (issue: #49095)
- Fixes potential memory corruption when determining seasonality #852
-
Prevent
prediction_field_name
clashing with other fields in machine learning results #861 - Include out-of-order as well as in-order terms in categorization reverse searches #950 (issue: #949)
- Mapping
-
- Ensure that field collapsing works with field aliases. #50722 (issues: #32648, #50121)
-
Improve DateFieldMapper
ignore_malformed
handling #50090 (issues: #46675, #50081) - Annotated text type should extend TextFieldType #49555 (issue: #49289)
- Ensure parameters are updated when merging flattened mappings. #48971 (issue: #48907)
- Network
-
- Fix TransportMasterNodeAction not Retrying NodeClosedException #51325
- Percolator
- Recovery
- Reindex
- SQL
-
- SQL: Fix milliseconds handling in intervals #51675 (issue: #41635)
- SQL: Fix ORDER BY YEAR() function #51562 (issue: #51224)
- SQL: change the way unsupported data types fields are handled #50823
- SQL: Optimisation fixes for conjunction merges #50703 (issue: #49637)
- SQL: Fix issue with CAST and NULL checking. #50371 (issue: #50191)
- SQL: fix NPE for JdbcResultSet.getDate(param, Calendar) calls #50184 (issue: #50174)
- SQL: COUNT DISTINCT returns 0 instead of NULL for no matching docs #50037 (issue: #50013)
- Fix LOCATE function optional parameter handling #49666 (issue: #49557)
- Fix NULL handling for FLOOR and CEIL functions #49644 (issue: #49556)
- Handle NULL arithmetic operations with INTERVALs #49633 (issue: #49297)
- Fix issue with GROUP BY YEAR() #49559 (issue: #49386)
- Fix issue with CASE/IIF pre-calculating results #49553 (issue: #49388)
- Fix issue with folding of CASE/IIF #49449 (issue: #49387)
- Fix issues with WEEK/ISO_WEEK/DATEDIFF #49405 (issues: #48209, #49376)
- SQL: Fix issue with mins & hours for DATEDIFF #49252
- SQL: Failing Group By queries due to different ExpressionIds #43072 (issues: #33361, #34543, #36074, #37044, #40001, #40240, #41159, #42041, #46316)
- Search
-
- Fix upgrade of custom similarity #50851 (issue: #50763)
- Fix NPE bug inner_hits #50709 (issue: #50539)
- Collect results in a local list when notifying partial results #49828 (issue: #49778)
- Fixes a bug in interval filter serialization #49793 (issue: #49519)
- Correctly handle duplicates in unordered interval matching #49775
- Correct rewritting of script_score query #48425 (issue: #48081)
- Do not throw errors on unknown types in SearchAfterBuilder #48147 (issue: #48074)
- Security
- Snapshot/Restore
-
- Fix Overly Aggressive Request DeDuplication #51270 (issue: #51253)
- Guard Repository#getRepositoryData for exception throw #50970
- Fix Index Deletion During Partial Snapshot Create #50234 (issues: #50200, #50202)
- Fix Index Deletion during Snapshot Finalization #50202 (issues: #45689, #50200)
- Fix RepoCleanup not Removed on Master-Failover #49217
- Make FsBlobContainer Listing Resilient to Concurrent Modifications #49142 (issue: #37581)
- Fix SnapshotShardStatus Reporting for Failed Shard #48556 (issue: #48526)
- Cleanup Concurrent RepositoryData Loading #48329 (issue: #48122)
- Transform
-
- Fix mapping deduction for scaled_float #51990 (issue: #51780)
- Fix stats can return old state information if security is enabled #51732 (issue: #51728)
- Fail to start/put on missing pipeline #50701 (issue: #50135)
- Fix possible audit logging disappearance after rolling upgrade #49731 (issue: #49730)
- Do not fail checkpoint creation due to global checkpoint mismatch #48423 (issue: #48379)