NOTE: You are looking at documentation for an older release. For the latest information, see the current release documentation.
Elasticsearch version 6.7.0
editElasticsearch version 6.7.0
editAlso see Breaking changes in 6.7.
Known issues
editA new docker
build_type
value was introduced in 6.7.0 for Docker images,
which caused two problems:
- If you perform a rolling upgrade to 6.7.0 on nodes that are running the official Docker container, they cannot rejoin the cluster. Upgrade to 6.7.1 instead or perform a full cluster restart upgrade. (issue: #40784)
-
If you have a 6.6 or earlier Java High Level REST client, it cannot understand the new
build_type
value. For example, it cannot read main responses on/
from a cluster that’s running 6.7.0. Upgrade your client to 6.7.0 or later. (issue: #40511)- Features/ILM
-
If an index is configured with an
index.lifecycle.name
that refers to a policy that does not exist, when index lifecycle management is stopped, index lifecycle management’s status will remain asSTOPPING
until all policies that do not exist are removed from any indices by using the Remove Policy API. (issue: #40824) - Deprecation Logging
- In a mixed cluster environment where one of the nodes is pre 6.7 but coordinating node is already upgraded to 6.7 an exception will be thrown when a deprecation warning is emitted on pre 6.7 node. it was introduced by (PR: #37622) in order to speed up the performance of header warning parsing. (issue: #60561)
Breaking changes
edit- Authentication
-
- The TokenService no longer accepts tokens generated by Elasticsearch 6.1 or earlier #38881
- CCR
- Distributed
- Infra/Packaging
- Ranking
Breaking Java changes
editDeprecations
edit- Analysis
- Audit
- CRUD
- Cluster Coordination
- Features/Features
- Features/Java High Level REST Client
- Features/Java Low Level REST Client
- Features/Watcher
-
- Deprecate Hipchat Watcher actions #39160
- Infra/Core
-
- Core: Deprecate negative epoch timestamps #36793
- Infra/Packaging
-
- Deprecate fallback to java on PATH #37990
- Infra/REST API
- Machine Learning
- Mapping
- Network
- Security
New features
editEnhancements
edit- Aggregations
-
- Add Composite to AggregationBuilders #38207 (issue: #38020)
- Allow nested fields in the composite aggregation #37178 (issue: #28611)
- Remove single shard optimization when suggesting shard_size #37041 (issue: #32125)
- Use List instead of priority queue for stable sorting in bucket sort aggregator #36748 (issue: #36322)
- Keys are compared in BucketSortPipelineAggregation so making key type… #36407
- Audit
- Authentication
- Authorization
- CCR
-
- Reduce retention lease sync intervals #40302
- Introduce forget follower API #39718 (issue: #37165)
- Renew retention leases while following #39335 (issues: #37165, #38718)
- Reduce refresh when lookup term in FollowingEngine #39184
- Integrate retention leases to recovery from remote #38829 (issue: #37165)
- Enable removal of retention leases #38751 (issue: #37165)
- Concurrent file chunk fetching for CCR restore #38495
- Tighten mapping syncing in ccr remote restore #38071 (issues: #36879, #37887)
- Do not allow put mapping on follower #37675 (issue: #30086)
- Added ccr to xpack usage infrastructure #37256 (issue: #37221)
- [CCR] FollowingEngine should fail with 403 if operation has no seqno assigned #37213
- [CCR] Added auto_follow_exception.timestamp field to auto follow stats #36947
- CRUD
-
- Add Seq# based optimistic concurrency control to UpdateRequest #37872 (issues: #10708, #36148)
- Introduce ssl settings to reindex from remote #37527 (issues: #29755, #37287)
- Use Sequence number powered OCC for processing updates #37308 (issues: #10708, #36148)
- Document Seq No powered optimistic concurrency control #37284 (issues: #10708, #36148)
- Enable IPv6 URIs in reindex from remote #36874
- Set acking timeout to 0 on dynamic mapping update #31140 (issues: #30672, #30844)
- Client
-
- Fixed required fields and paths list #39358
- Cluster Coordination
- Distributed
-
- Add BWC for retention leases #39482 (issue: #37165)
- Allow retention lease operations under blocks #39089 (issues: #34648, #37165)
- Remove retention leases when unfollowing #39088 (issues: #34648, #37165)
- Introduce retention lease state file #39004 (issues: #37165, #38588, #39032)
- Introduce retention lease actions #38756 (issue: #37165)
- Add dedicated retention lease exceptions #38754 (issue: #37165)
- Recover retention leases during peer recovery #38435 (issue: #37165)
- Lift retention lease expiration to index shard #38380 (issues: #37165, #37963, #38070)
- Introduce retention lease background sync #38262 (issue: #37165)
- Copy retention leases when trim unsafe commits #37995 (issue: #37165)
- Expose retention leases in shard stats #37991 (issue: #37165)
- Introduce retention leases versioning #37951 (issue: #37165)
- Soft-deletes policy should always fetch latest leases #37940 (issues: #37165, #37375)
- Sync retention leases on expiration #37902 (issue: #37165)
- Ignore shard started requests when primary term does not match #37899 (issue: #33888)
- Move update and delete by query to use seq# for optimistic concurrency control #37857 (issues: #10708, #36148, #37639)
- Introduce retention lease serialization #37447 (issues: #37165, #37398)
- Add run under primary permit method #37440 (issue: #37398)
- Introduce retention lease syncing #37398 (issue: #37165)
- Introduce retention lease persistence #37375 (issue: #37165)
- Add validation for retention lease construction #37312 (issue: #37165)
- Introduce retention lease expiration #37195 (issue: #37165)
- Introduce shard history retention leases #37167 (issue: #37165)
- Always initialize the global checkpoint #34381
- Docs Infrastructure
- Engine
-
- Also mmap cfs files for hybridfs #38940 (issue: #36668)
- Specialize pre-closing checks for engine implementations #38702
- Ensure that max seq # is equal to the global checkpoint when creating ReadOnlyEngines #37426
- Enable Bulk-Merge if all source remains #37269
- Introduce time-based retention policy for soft-deletes #34943 (issue: #34908)
- Features/CAT APIs
- Features/Features
- Features/ILM
-
- Ensure ILM policies run safely on leader indices #38140 (issue: #34648)
- Skip Shrink when numberOfShards not changed #37953 (issue: #33275)
- Inject Unfollow before Rollover and Shrink #37625 (issue: #34648)
- Add set_priority action to ILM #37397 (issue: #36905)
- [ILM] Add Freeze Action #36910 (issue: #34630)
- Features/Indices APIs
-
- New mapping signature and mapping string source fixed. #37401
- Features/Ingest
-
- minor updates for user-agent ecs for 6.7 #39213 (issue: #38757)
- Dep. check for ECS changes to User Agent processor #38362 (issue: #36024)
- Add ECS schema for user-agent ingest processor #37727 (issue: #37329)
- ingest: compile mustache template only if field includes {{' #37207 (issue: #37120)
- Move ingest-geoip default databases out of config #36949 (issue: #36898)
- Features/Java High Level REST Client
- Features/Watcher
- Infra/Core
-
- Use DateFormatter in monitoring instead of joda code #38309
- Use dateformatter in ingest-common to log deprecations #38099
- Add simple method to write collection of writeables #37448 (issue: #37398)
- Date/Time parsing: Use java time API instead of exception handling #37222
- [API] spelling: interruptible #37049 (issue: #37035)
- restrict node start-up when cluster name in data path #36519 (issue: #32661)
- Infra/Logging
-
- Trim the JSON source in indexing slow logs #38081 (issue: #38080)
- Optimize warning header de-duplication #37725 (issues: #35754, #37530, #37597, #37622)
- Remove warn-date from warning headers #37622 (issues: #35754, #37530, #37597)
- Add some deprecation optimizations #37597 (issues: #35754, #37530)
- Only update response headers if we have a new one #37590 (issues: #35754, #37530)
- Infra/Packaging
- Infra/Scripting
-
- Add getZone to JodaCompatibleZonedDateTime #37084
- Infra/Settings
- License
- Machine Learning
-
- Allow stop unassigned datafeed and relax unset upgrade mode wait #39034
- Move ML Optimistic Concurrency Control to Seq No #38278 (issues: #10708, #36148)
- Add upgrade mode docs, hlrc, and fix bug #37942
- Tighten up use of aliases rather than concrete indices #37874
- Add support for single bucket aggs in Datafeeds #37544 (issue: #36838)
- Migrate unallocated jobs and datafeeds #37536 (issue: #32905)
- Adjust seccomp filter for Fedora 29. #354
- Mapping
-
- Only issue a deprecation warning if include_type_name is not set. #38825 (issue: #35190)
- Log document id when MapperParsingException occurs #37800 (issue: #37658)
- Types removal - add constants for include_type_names #37304
- Deprecation check for index_options on numeric fields #37026 (issue: #36024)
- Deprecation check for indices with multiple types #36952 (issues: #35190, #36024)
- Use index-prefix fields for terms of length min_chars - 1 #36703
- Recovery
-
- Do not wait for advancement of checkpoint in recovery #39006 (issues: #38949, #39000)
- SyncedFlushService.getShardRoutingTable() should use metadata to check for index existence #37691 (issue: #33888)
- Make prepare engine step of recovery source non-blocking #37573 (issue: #37174)
- Make recovery source send operations non-blocking #37503 (issue: #37458)
- Prepare to make send translog of recovery non-blocking #37458 (issue: #37291)
- Make finalize step of recovery source non-blocking #37388 (issue: #37291)
- Make recovery source partially non-blocking #37291 (issue: #36195)
- Do not mutate RecoveryResponse #37204 (issue: #37174)
- Don’t block on peer recovery on the target side #37076 (issue: #36195)
- Reduce recovery time with compress or secure transport #36981 (issue: #33844)
- Rollup
-
- Replace the TreeMap in the composite aggregation #36675
- SQL
-
- SQL: Enhance checks for inexact fields #39427 (issue: #38501)
- SQL: change the default precision for CURRENT_TIMESTAMP function #39391 (issue: #39288)
- SQL: add "validate.properties" property to JDBC’s allowed list of settings #39050 (issue: #38068)
- SQL: Allow look-ahead resolution of aliases for WHERE clause #38450 (issue: #29983)
- SQL: Implement CURRENT_DATE #38175 (issue: #38160)
- SQL: Generate relevant error message when grouping functions are not used in GROUP BY #38017 (issue: #37952)
- SQL: Skip the nested and object field types in case of an ODBC request #37948 (issue: #37801)
- SQL: Add protocol tests and remove jdbc_type from drivers response #37516 (issues: #36635, #36882)
- SQL: Remove slightly used meta commands #37506 (issue: #37409)
- SQL: Describe aliases as views #37496 (issue: #37422)
-
SQL: Make
FULL
non-reserved keyword in the grammar #37377 (issue: #37376) - SQL: Use declared source for error messages #37161
- SQL: Improve error message when unable to translate to ES query DSL #37129 (issue: #37040)
- [API] spelling: subtract #37055 (issue: #37035)
- [API] spelling: similar #37054 (issue: #37035)
- [API] spelling: input #37048 (issue: #37035)
- SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg #36933 (issue: #36903)
- SQL: Preserve original source for each expression #36912 (issue: #36894)
- Search
-
- Add finalReduce flag to SearchRequest #38104 (issues: #37000, #37838)
- Expose sequence number and primary terms in search responses #37639
- Allow field types to optimize phrase prefix queries #37436 (issue: #31921)
- Add support for providing absolute start time to SearchRequest #37142 (issue: #32125)
- Ensure that local cluster alias is never treated as remote #37121 (issues: #32125, #36997)
- [API] spelling: cacheable #37047 (issue: #37035)
- Add ability to suggest shard_size on coord node rewrite #37017 (issues: #32125, #36997, #37000)
- Skip final reduction if SearchRequest holds a cluster alias #37000 (issues: #32125, #36997)
- Add support for local cluster alias to SearchRequest #36997 (issue: #32125)
- Security
- Snapshot/Restore
-
- RestoreService should update primary terms when restoring shards of existing indices #38177 (issue: #33888)
- Allow open indices to be restored #37733
- Create specific exception for when snapshots are in progress #37550 (issue: #37541)
- SNAPSHOT: Speed up HDFS Repository Writes #37069
- Implement Atomic Blob Writes for HDFS Repository #37066 (issue: #37011)
- [API] spelling: repositories #37053 (issue: #37035)
- SNAPSHOT: Use CancellableThreads to Abort #35901 (issue: #21759)
- Suggesters
Bug fixes
edit- Aggregations
-
- Skip sibling pipeline aggregators reduction during non-final reduce #40101 (issue: #40059)
- Only create MatrixStatsResults on final reduction #38130 (issue: #37587)
-
Don’t load global ordinals with the
map
execution_hint #37833 (issue: #37705) - Issue #37303 - Invalid variance fix #37384 (issue: #37303)
- Allocation
- Analysis
- Audit
- Authentication
-
- Correct authenticate response for API key #39684
- Fix security index auto-create and state recovery race #39582
- Use consistent view of realms for authentication #38815 (issue: #30301)
- Enhance parsing of StatusCode in SAML Responses #38628
- Limit token expiry to 1 hour maximum #38244
- Fix expired token message in Exception header #37196
- Fix NPE in CachingUsernamePasswordRealm #36953 (issue: #36951)
- CCR
-
- Safe publication of AutoFollowCoordinator #40153 (issue: #38560)
- Enable reading auto-follow patterns from x-content #40130 (issue: #40128)
- Stop auto-followers on shutdown #40124
- Protect against the leader index being removed #39351 (issue: #39308)
- Fix shard follow task startup error handling #39053 (issue: #38779)
- Filter out upgraded version index settings when starting index following #38838 (issue: #38835)
-
Handle the fact that
ShardStats
instance may have no commit or seqno stats #38782 (issue: #38779) - Fix LocalIndexFollowingIT#testRemoveRemoteConnection() test #38709 (issue: #38695)
- Prevent CCR recovery from missing documents #38237
- Fix file reading in ccr restore service #38117
- Correct argument names in update mapping/settings from leader #38063
- Ensure changes requests return the latest mapping version #37633
- Do not set fatal exception when shard follow task is stopped. #37603
- Add fatal_exception field for ccr stats in monitoring mapping #37563
- Do not add index event listener if CCR disabled #37432
- When removing an AutoFollower also mark it as removed. #37402 (issue: #36761)
- [CCR] Resume follow Api should not require a request body #37217 (issue: #37022)
- CRUD
- Cluster Coordination
- Distributed
-
- Enforce retention leases require soft deletes #39922 (issue: #39914)
- Treat TransportService stopped error as node is closing #39800 (issue: #39584)
- Use cause to determine if node with primary is closing #39723 (issue: #39584)
- Don’t ack if unable to remove failing replica #39584 (issue: #39467)
- Ignore waitForActiveShards when syncing leases #39224 (issue: #39089)
- Fix synchronization in LocalCheckpointTracker#contains #38755 (issues: #33871, #38633)
- TransportVerifyShardBeforeCloseAction should force a flush #38401 (issues: #33888, #37961)
- Fix limit on retaining sequence number #37992 (issue: #37165)
- Close Index API should force a flush if a sync is needed #37961 (issues: #33888, #37426)
- Force Refresh Listeners when Acquiring all Operation Permits #36835
- Replaced the word shards with replicas in an error message. (#36234) #36275 (issue: #36234)
- Engine
- Features/Features
-
- Only count some fields types for deprecation check #40166
- Deprecation check for indices with very large numbers of fields #39869 (issue: #39851)
- Check for .watches that wasn’t upgraded properly #39609
- Link to 7.0 documentation in deprecation checks #39194
- Handle Null in FetchSourceContext#fetchSource #36839 (issue: #29293)
- Features/ILM
-
- Handle failure to release retention leases in ILM #39281 (issue: #39181)
- Preserve ILM operation mode when creating new lifecycles #38134 (issues: #38229, #38230)
- Retry ILM steps that fail due to SnapshotInProgressException #37624 (issues: #37541, #37552)
-
Remove
indexing_complete
when removing policy #36620
- Features/Indices APIs
- Features/Ingest
- Features/Java High Level REST Client
-
-
Allow setting of
copy_settings
in the HLRC #39752 (issue: #30255) - Update IndexTemplateMetaData to allow unknown fields #38448 (issue: #36938)
-
if_seq_no
andif_primary_term
parameters aren’t wired correctly in REST Client’s CRUD API #38411 - Update Rollup Caps to allow unknown fields #38339 (issue: #36938)
- Fix ILM explain response to allow unknown fields #38054 (issue: #36938)
- Fix ILM status to allow unknown fields #38043 (issue: #36938)
- Fix ILM Lifecycle Policy to allow unknown fields #38041 (issue: #36938)
- Update authenticate to allow unknown fields #37713 (issue: #36938)
- Update verify repository to allow unknown fields #37619 (issue: #36938)
- Update get users to allow unknown fields #37593 (issue: #36938)
- Update Execute Watch to allow unknown fields #37498 (issue: #36938)
- Update Put Watch to allow unknown fields #37494 (issue: #36938)
- Update Delete Watch to allow unknown fields #37435 (issue: #36938)
- Fix weighted_avg parser not found for RestHighLevelClient #37027 (issue: #36861)
-
Allow setting of
- Features/Monitoring
- Features/Watcher
- Geo
- Highlighting
- Infra/Core
-
- Correct name of basic_date_time_no_millis #39367
- Fix DateFormatters.parseMillis when no timezone is given #39100 (issue: #39067)
- Prefix java formatter patterns with 8 #38712 (issue: #38567)
- Bubble-up exceptions from scheduler #38317 (issue: #38014)
- Core: Revert back to joda’s multi date formatters #36814 (issues: #36447, #36602)
- Propagate Errors in executors to uncaught exception handler #36137 (issue: #28667)
- Infra/Packaging
- Infra/Scripting
-
- Fix Painless void return bug #38046
- Infra/Settings
- Machine Learning
-
- Fix race condition when creating multiple jobs #40049 (issue: #38785)
- Fix datafeed skipping first bucket after lookback when aggs are used #39859 (issue: #39842)
- Refactoring lazy query and agg parsing #39776 (issue: #39528)
- Allow aliased .ml-anomalies* index on PUT Job #38821 (issue: #38773)
- Report index unavailable instead of waiting for lazy node #38423
- Prevent submit after autodetect worker is stopped #37700 (issue: #37108)
- Fix ML datafeed CCS with wildcarded cluster name #37470 (issue: #36228)
- Update error message for process update #37363
- Make GetJobStats work with arbitrary wildcards and groups #36683 (issue: #34745)
- Mapping
-
- Make sure to reject mappings with type _doc when include_type_name is false. #38270 (issue: #38266)
-
Treat put-mapping calls with
_doc
as a top-level key as typed calls. #38032 - Update the deprecation message for typed put mapping requests. #37835
- Make sure PutMappingRequest accepts content types other than JSON. #37720
- MAPPING: Improve Precision for scaled_float #37169 (issue: #32570)
- Make sure to accept empty unnested mappings in create index requests. #37089
- Stop automatically nesting mappings in index creation requests. #36924
- Network
- Ranking
-
- QueryRescorer should keep the window size when rewriting #36836
- Recovery
- SQL
-
- SQL: Preserve original source for cast/convert function #40271 (issue: #40239)
- SQL: Fix issue with optimization on queries with ORDER BY/LIMIT #40256 (issue: #40211)
- SQL: Fix issue with getting DATE type in JDBC #40207
- SQL: Fix issue with date columns returned always in UTC #40163 (issue: #40152)
- SQL: Add multi_value_field_leniency inside FieldHitExtractor #40113 (issue: #39700)
- SQL: fix incorrect ordering of groupings (GROUP BY) based on orderings (ORDER BY) #40087 (issue: #39956)
- SQL: Fix bug with JDBC timezone setting and DATE type #39978 (issue: #39915)
- SQL: Wrap ZonedDateTime parameters inside scripts #39911 (issue: #39877)
- SQL: ConstantProcessor can now handle NamedWriteable #39876 (issue: #39875)
- SQL: Extend the multi dot field notation extraction to lists of values #39823 (issue: #39738)
- SQL: values in datetime script aggs should be treated as long #39773 (issue: #37042)
- SQL: Don’t allow inexact fields for MIN/MAX #39563 (issue: #39427)
- SQL: Fix merging of incompatible multi-fields #39560 (issue: #39547)
- SQL: fix COUNT DISTINCT column name #39537 (issue: #39511)
- SQL: ignore UNSUPPORTED fields for JDBC and ODBC modes in SYS COLUMNS #39518 (issue: #39471)
- SQL: Use underlying exact field for LIKE/RLIKE #39443 (issue: #39442)
- SQL: enforce JDBC driver - ES server version parity #38972 (issue: #38775)
- SQL: fall back to using the field name for column label #38842 (issue: #38831)
- SQL: Prevent grouping over grouping functions #38649 (issue: #38308)
- SQL: Relax StackOverflow circuit breaker for constants #38572 (issue: #38571)
- SQL: Fix issue with IN not resolving to underlying keyword field #38440 (issue: #38424)
- SQL: change the Intervals milliseconds precision to 3 digits #38297 (issue: #37423)
- SQL: Fix esType for DATETIME/DATE and INTERVALS #38179 (issue: #38051)
- SQL: Added SSL configuration options tests #37875 (issue: #37711)
- SQL: Fix casting from date to numeric type to use millis #37869 (issue: #37655)
- SQL: Fix BasicFormatter NPE #37804
- SQL: Return Intervals in SQL format for CLI #37602 (issues: #29970, #36186, #36432)
- SQL: fix object extraction from sources #37502 (issue: #37364)
- SQL: Fix issue with field names containing "." #37364 (issue: #37128)
- SQL: Fix bug regarding alias fields with dots #37279 (issue: #37224)
- SQL: Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) #37254 (issue: #30285)
- SQL: fix COUNT DISTINCT filtering #37176 (issue: #37086)
- SQL: Fix issue with wrong NULL optimization #37124 (issue: #35872)
- SQL: Fix issue with complex expression as args of PERCENTILE/_RANK #37102 (issue: #37099)
- SQL: Handle the bwc Joda ZonedDateTime scripting class in Painless #37024 (issue: #37023)
- SQL: Fix bug regarding histograms usage in scripting #36866
- SQL: Fix issue with always false filter involving functions #36830 (issue: #35980)
- SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests #36800 (issue: #36756)
- SQL: Enhance Verifier to prevent aggregate or grouping functions from #36799 (issue: #36798)
- SQL: normalized keywords shouldn’t be allowed for groupings and sorting [ISSUE] #35203
- Search
-
- Serialize top-level pipeline aggs as part of InternalAggregations #40177 (issues: #40059, #40101)
- Fix Fuzziness#asDistance(String) #39643 (issue: #39614)
- Fix simple query string serialization conditional #38960 (issues: #21504, #38889)
- Ensure that maxConcurrentShardRequests is never defaulted to 0 #38734
- Look up connection using the right cluster alias when releasing contexts #38570
- Fix fetch source option in expand search phase #37908 (issue: #23829)
- Throw if two inner_hits have the same name #37645 (issue: #37584)
- Ensure either success or failure path for SearchOperationListener is called #37467 (issue: #37185)
-
Use executor
SAME
to handle search related handlers #37427 (issues: #33732, #37392)
- Security
- Snapshot/Restore
-
- Fix Concurrent Snapshot Ending And Stabilize Snapshot Finalization #38368 (issue: #38226)
- Fix Two Races that Lead to Stuck Snapshots #37686 (issues: #32265, #32348)
- Fix Race in Concurrent Snapshot Delete and Create #37612 (issue: #37581)
- Streamline S3 Repository- and Client-Settings #37393
- SNAPSHOTS: Upgrade GCS Dependencies to 1.55.0 #36634 (issues: #35229, #35459)
- Suggesters
- Task Management
-
- Un-assign persistent tasks as nodes exit the cluster #37656