What’s new in 8.11

edit

What’s new in 8.11

edit

Here are the highlights of what’s new and improved in 8.11. For detailed information about this release, check the release notes.

Previous versions: 8.10 | 8.9 | 8.8 | 8.7 | 8.6 | 8.5 | 8.4 | 8.3 | 8.2 | 8.1 | 8.0

Discover

edit

ES|QL in Discover

edit

[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. In 8.11 we are introducing Elasticsearch Query Language (ES|QL), Elastic’s new piped language for data exploration and investigation. ES|QL transforms, enriches, and simplifies your data exploration process.

A short video demo of ES|QL

Here is what you can expect:

  • Easy start: To begin using ES|QL in Discover, select Try ES|QL from the data view menu.
  • Efficient and easy query building: ES|QL in Discover offers auto-complete and in-app documentation, making it easy to craft powerful queries right from the query bar.
  • Comprehensive and powerful data exploration: Conduct ad-hoc data exploration within Discover. Create aggregations, transform data, enrich datasets, and more directly from the query builder. Results are presented in a tabular format or as visualizations. It depends on the query you are executing.
  • Contextual visualizations: When writing ES|QL queries in Discover, you’ll receive visual representations powered by the Lens suggestion engine. Your query’s nature determines the type of visualization you get, such as a metric, histogram, heatmap, and so on.
  • Enrichment: Use the enrich command to enhance your query dataset with fields from another dataset, complete with in-context suggestions for the selected policy, such as hinting the matching field and enriched columns.
  • In-line visualization editing: Edit ES|QL visualizations directly within Discover and dashboards. No need to navigate to Lens for quick edits, so you can make changes seamlessly.
  • Dashboard integration: Save your ES|QL visualizations to a dashboard directly from Discover once you’re satisfied with the results.
  • Alerting: Use ES|QL for Observability and Security alerts, setting aggregated values as thresholds. Enhance detection accuracy and receive actionable notifications by emphasizing meaningful trends over isolated incidents, reducing false positives.

ES|QL in Discover brings efficiency and power to your data investigations, streamlining your path to insights.

Learn more about ES|QL’s capabilities in the Elasticsearch ES|QL documentation.

An example of ES|QL
An example using ES|QL in an Observability use case

Dashboard

edit

Lens inline editing in dashboards

edit

You can now edit a Lens visualization without leaving the dashboard instead of navigating back and forth to the Lens editor. Open the panel menu and select Edit visualization. A flyout will be open in the dashboard where you can perform any edits to your Lens panels. Once happy with your edits, click Apply and close. This new editing experience is more convenient and will save you time since the dashboard will not need to reload when saving your changes.

An example of inline editing in Lens

Links panel

edit

You can now easily navigate from one dashboard to another using the links panel. Better organize your dashboards and make them more performant by chunking them in multiple dashboards with fewer visualizations and linking them together. You can carry over your filters, query, and time range when navigating to other related dashboards. Display your links horizontally or vertically as it better suits your dashboard layout. You can also use the links panel to include external links in your dashboards, such as to your wiki page or other applications. Decide whether you want to open the links in the same browser tab or in a new one.

An example of the new links panel
An example of the edit links panel flyout
An example of how to edit links using the panel flyout

Color mapping for enhanced data visualization

edit

Color is a fundamental visual element, alongside position and shape, that plays a crucial role in conveying information effectively.

An example of the new color mapping feature

Here’s what our new color mapping feature offers:

  • Effortless categorization: Easily assign one or more field categories to specific colors. This makes it simpler than ever to organize and understand your data.
  • Guided color selection: Our intuitive color chooser provides you with predefined palettes that ensure your charts not only look great, but also align seamlessly with different Kibana themes.
  • Enhanced color palettes: We’ve reintroduced a clear concept of color palettes, making it easier for you to select and apply gradients, improving the overall aesthetics of your visualizations.

Whether you’re working with cartesian, partition, or tag clouds charts, these enhancements are designed to help you make the most of your data. With this feature, you can expect an improved ability to categorize, differentiate, and emphasize data points on your charts, ultimately leading to better insights and more visually appealing dashboards.

Cross-cluster search (CCS) query inspector

edit

Customers querying data from multiple clusters (CCS queries) will get more information about each of the cluster’s responses. For each of the visualizations in a dashboard as well as in Discover, you can look at response times per cluster and shard failures. This is especially important when one or more clusters are not able to provide all the data, so you know that you are looking at partial results and why they fail.

An example of the new cross-cluster search query inspector

Individual annotation editing from the library

edit

We’ve introduced full annotation group editing in the Visualize Library. Now, you can easily edit shared annotation groups without leaving Lens. No more searching for consuming visualizations.

An example of annotation editing

Machine Learning

edit

ELSER is improved and is now generally available

edit

In 8.8, we introduced Elastic Learned Sparse Encoder in technical preview. ELSER is Elastic’s text expansion language model for AI search. It offers superior relevance out of the box, without the need for retraining on in-domain data or any other ML or MLOps effort. Deploy it with a couple of clicks from Elastic’s UI and start leveraging the power of AI with your search.

In 8.11, we’re releasing a generally available second version. ELSER model-2 comes in two versions:

  • The optimized model, which runs on the linux-x86_64 platform
  • The cross-platform model

Both ELSER model-2 versions, platform-optimized and cross-platform, show improved relevance compared to the original ELSER release, as measured against the BEIR benchmarks. Importantly, the optimized version also shows significantly improved performance (reduced inference latency). The Elastic Cloud supports the optimized version and so Elastic Cloud users will benefit from the materially improved performance of the optimized ELSER model-2.

An example of ELSER model 2

The original version of ELSER (the model available prior to 8.11) will remain in technical preview.

Inference APIs

edit

[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. We are working to introduce a unified inference API that abstracts away the complexity of performing inference on different models that are trained for different tasks. The API introduces a simple, intuitive syntax of the form:

POST /_inference/<task_type>/<model_id>

In 8.11, we’re’releasing a contained first MVP iteration of this framework, which initially only supports ELSER. This greatly simplifies the syntax for creating an inference pipeline.

More importantly, in the future the new inference API will support both internal and external models and will integrate with the LLM ecosystem for our users to have the most powerful AI effortlessly and seamlessly at their fingertips, through a unified, self-explanatory API.

AIOps: Log rate analysis supports text fields

edit

[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Continuing enhancing log analysis capabilities with smart AIOps tools for drastically shorter mean time to repair. We now support detection of log rate changes that are due to text fields, for example the common message log field. Previously log rate change was limited to detecting spikes and dips caused by keyword fields. By adding text fields, we’ve incorporated pattern analysis into log rate analysis. That has significantly reduced the time it takes to detect and diagnose events that used to go unnoticed for periods of time.

An example of log rate analysis supporting text fields

Data drift workflows

edit

[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. In 8.10, we introduced the Data comparison view to help you detect data drift. In 8.11, we have renamed it to Data drift and we have enhanced it to include workflows that help you visualize changes in the model input data and detect potential model performance degradation over time.

An example of data drift workflows

Improvements in anomaly detection embeddability and data frame analytics pipelines

edit

We have improved the UX for attaching anomaly swim lanes and anomaly charts to dashboards. It is now more friendly and consistent with the same functionality from Lens. In addition, you can now attach these ML charts to new dashboards (previously this was only able to be done for existing ones).

In Data Frame Analytics, we previously added the ability to link directly to Discover and Dashboards from the results data grid filtering for the row’s field/values for all visible columns. For improved usability, you can now do this during the job creation as well. We have also made UX improvements for the deployment of trained models from Data Frame Analytics jobs, including an option to reindex your data at the end of the ingest pipeline creation.

ResponseOps

edit

New ES|QL rule type

edit

A new ES|QL alerting rule type is now available under the existing Elasticsearch rule type. This rule type brings all the new functionalities that are available within the new and powerful language, ES|QL, to Kibana Alerting to allow and unlock new alerting use cases.

With the new type, users will be able to generate a single alert based on defined ES|QL query and preview the query result before saving the rule. When the query returns an empty result no alerts will be generated.

An example of creating a rule type using ES|QL

Kibana cases custom field

edit

A new functionality is now available in Kibana cases. Users will be able to add custom fields to the case structure, so they can use it for better classification and case enrichment. As a first step, those fields will be available in the case view only. In the next step, we’re planning to have more field types, dedicated privileges, support filters, and search capabilities in the case table.

An example of creating a custom case field in Kibana

Supporting multi levels of term aggregations in Elasticsearch rule type

edit

The existing Elasticsearch alerting rule is now supported by multiple selection when grouping by alert fields, which allows you to define multiple layers of term aggregations.

An example of creating multiple layers of term aggregations

Slack connector - allow List

edit

The Slack connector supports a new allow list, so customers will be able to manage the available Slack channels within the alert actions.

Observability

edit

Elasticsearch alerting rule now available in Observability

edit

The Elasticsearch alerting rule is now available in Observability. Before this update, Observability customers who leverage the Elasticsearch rule were required to move between Stack Management and Observability to manage their alerts. To manage the Observability roles properly, we added a new field to determine which role can maintain the created rule and its generated alerts.

An example of creating an altering rule in Observability

Global Experience

edit

Create and manage ES|QL enrichment index policies

edit

In support of our new ES|QL capabilities, we have added enrich policies to our Index Management experience. Users can now create their enrichment policies right from here and get started using it right away. Once configured, all enrich policies are available in the Enrich Policies tab.

Example of an ES|QL Query using an enrich policy with enrich:

An example of an ES|QL query using an enrich policy

Example of the enrich policies shown in the UI:

An example of where to add an enrich policy in the UI
An example of where an enrich policy lives in the UI

Start Discover ES|QL from global search

edit

Discover searches are powered by KQL by default. You can easily switch to ES|QL mode within Discover and search your data with this new query language. You can also access ES|QL in Discover from the global search bar within Elastic. With just a few keystrokes, simply type in “ESQL” and you can access Discover with ES|QL enabled for you.

An example of using ES|QL in Discover

Access Elasticsearch connection details

edit

You can now view your Elasticsearch endpoint, Cloud ID, and even manage your API keys from many areas in Elastic such as integrations. You can also access the connection details on any page in Elastic using the help menu from the header bar.

An example of where to find the endpoints in the UI

AWS CloudFormation template updates

edit

When subscribing to Elastic using the AWS Marketplace, users have the option to quickly get set up with an AWS CloudFormation Template. This step allows you to create an Elastic deployment in the AWS region of your choice. We’ve updated the AWS CloudFormation Template with bug fixes and stability improvements to better help you get started.