Keeping up with Kibana: This week in Kibana for July 26th, 2019

By
Bill McConaghy

Kibana UI Update for July 22-26

Hiring

Security

Copy to Space

Work continues on the copy-to-space initiative. The API changes are just about ready for review, and have a full suite of functional tests. Additionally, we have changes to the core platform underway to support this feature.

Kerberos 

We found and fixed a bug in the Kerberos auth provider when the client requests mutual authentication and when the negotiation process takes multiple steps before completion. We’ve also been working with Cloud to ensure that Kerberos works properly in ECE and ESS. After working through a number of issues, the last missing piece is a change to the allocator’s health check of Kibana to work with the 401s that are returned when using the Kerberos auth provider.

Platform

New Platform 

We merged HTTP route handler RFC and started working on the initial implementation. This is one of the final pieces need to allow NP plugins to register HTTP endpoints.

Work has moved forward on the AppService. One blocker, the ContextService, is nearing final review. In addition, some of the kinks of how legacy platform “hacks” will run within the context of a New Platform app have been solved. This was the final roadblock to a working  end-to-end AppService.

We’ve begun tracking plugin migration work more closely. We ask that each team add the “Feature: NP Migration” label to all migration-related PRs in GitHub going forward. We’ll be using this to report on migration progress starting next week.

Operations

We’ve made pretty good progress into the static filesystem last week. Tests were completed, a build, test and deploy pipeline was achieved across multiple environments and finally, there is also documentation for the project right now. 

We have a PR in progress which will reduce the bootstrap time by 15-25 seconds by committing the package builds to the repository.

We continue planning out the improvements to our CI pipeline, which is now being tracked here. There is a lot of duplicated effort done in our CI jobs which further introduces space for random failures. The goal here is to create multi-tiered workers so the output of one can be the input for another while also allowing for faster re-builds should a flaky failure happen.

Geo-Maps App

EMS Vector Tiles in Maps app (7.4) 

Work started on pulling in EMS vector tles in the Maps app. This will be the new default, instead of our current raster maps (#20791).

Custom color ramps (7.4)

Probably the most requested feature for Maps. In progress 

Pew pew maps for SIEM (7.4)

Working on POC to integrate pew pew maps with SIEM. We want this to work at all scales, a global overview when zoomed out, as well as show request details when zoomed in. (41504)

And more

  • Bug fixes 7.3 and up
    • Remove old data from map when there’s error in most recent query #41358
    • Allow joins across data-types (e.g. string to integer) #41503
    • Fix error on startup #42073
    • Fit only on visible features #42020
    • Fix tooltips for documents with numerical doc-ids #42011
    • Show ids in tooltips #41684
    • Improve geojson upload performance #41540
  • Improvements
    • Improve look&feel of tooltips #41793
    • Show indication of incomplete data in Layer TOC #41271
  • Housekeeping (no direct end-user impact)

Geo-Elastic Maps Service

All first level administrative boundaries of countries in South America are now published for all versions of Kibana, each with ISO and local identifiers. This brings the total number of EMS layers now to 48. See maps.elastic.co for details. And of course, use Kibana to pull in the layer for your custom Maps.

App Architecture

We've continued our bug cleanup efforts, covering issues related to inverting disabled filters, duplicate fetches triggered by filters, unexpected warnings when creating index patterns, and tagcloud errors with significant terms aggs.

Data Access Services

We opened a PR to cancel visualize fetches when navigating away or re-fetching, which closes 4 issues.

New Platform Migration

We've continued moving full-steam ahead on migration efforts, including more Angular cleanup in top_nav, updates to the markdown vis type, and shims for tile map, region map, and table visualizations.

A first pass is almost done on TypeScripting index patterns to prep for migration, and we are in the early stages of shimming agg_types and agg_configs.

In an effort to share what we've been learning with the rest of the team, we took some time to merge a few updates to the migration guide, particularly around shimming browser-side plugins. We're expecting to schedule a knowledge sharing session on these topics sometime next week… more details to follow in the next UI weekly update.

Kibana App (Visualizations, Sharing, Discover)

Lens (New visual editor)

This week we focused on improving the UX for creating Layers, which is better able to support multiple chart types now:

We also merged our Embeddable PR which allows embedding a Lens visualization in Dashboards with support for dashboard filtering.

Next week we are going to be working on:

  • Sharing our progress with the QA team to prepare for our beta release
  • Building an improved chart switching UI
  • Making suggestions smarter
  • Integrating the filter bar and time picker into the Lens app

Elastic Charts

Most of the work was focused on bringing a shared API to include new chart types. We have a simplified interface to create spec components and we are trying to move away from Mobx state to a Redux one that will help us to split the state in two: the chart specification state (every configuration build upon the react spec components) and the chart type specific state (that will be created and used after the configuration are parsed). We were also working on fixing the last bug to replace the Discover chart. We worked on adding the ability to color bars  depending on a color function passed to the chart configuration. We finally merged also the empty chart state PR that introduce a standard message for the user when the Chart is empty. 

Discover

Shard failures

Also we heard that Support really does not find the "1 of 2 shards failed" warning toast helpful at all. So we implemented a proper shard failure error dialog, that you can open from the warning toast and inspect every shard failure in detail. Yay, no more HAR recordings for shard failures … from 7.4 onwards (PR #41649 will be merged soon). This modal will work in Discover, Visualize and Dashboard for shard failures.

Canvas

Box select

Lets the user highlight an area to then move, rotate or group the elements that are enclosed by the box (#32995) - this is included in 7.3, but wasn’t really highlighted until now.  It’s an important add, and one that will make designing with Canvas easier and more expressive.

Design

Elastic Chart snippets coming to EUI 

We’re making progress both on providing some basic styles for Elastic Charts and for providing documentation within EUI to give you very easy copy/paste snippets for design team approved™ charts. Storybook will still continue to be a reference for the deep divers, but if you just need something simple that looks good, we plan on having lots of options.

EUI Data Grid updates

We’re now working off of a feature branch for the data grid component. Engineering and design are working together to build up the features over the next couple weeks. Right now we have column resizing and some customization around styling. Expect this one to move quickly over the next few weeks. Our goal is still to get this available to folks by the end of August.

EuiSuggest components progress

We’re working with the App team to add saved queries to the KQL search bar. As part of that we’re making official EUI components and patterns for how to manage the search bar. These will just be patterns and visual components.

Stack Services

Alerting Service

This week changes to the task manager for better error handling has been merged (#39829). This PR converts task manager to use saved objects and allows to define custom error handling when a task failed. Now that the task manager supports more custom error handling, work has begun making alerting use the new features (#41917).

Work on the API Keys has begun, there’s an open PR to enable SSL for the integration tests whenever security is enabled (#41765). This will allow us to write integration tests for alerting when security is enabled (using API Keys which require SSL) as well as when security is disabled (#41389).

Work has started on adding throttling support (#41919).

Not-yet-merged PR for @kbn/config-schema to add a nullable()type. (#41728)

Alerting plugin converted from using joi to @kbn/config-schema for per-alert-type configuration object validation. (#41801)

New built-in action to index data into ES, for parity with watcher. (#41592)

Start of HTTP API cleanup to make endpoints similar in behavior with other Kibana endpoints; first up is cleaning up HTTP DELETE responses. (#41933)

Reporting

Work continues on our reporting handbook, which aims to help support and other engineers get familiar with all aspects of the reporting pipeline. We’re also making some improvements to our logging and debugging related infrastructure in hopes that this will improve customer’s ability to use our reporting capability. Finally, we’ve addressed a few bugs that our users noticed:

QA

Cloud Test Automation

  • Issues can be found here and require triage by relevant functional teams:

https://github.com/elastic/kibana/projects/20

Visual Test Automation

  • Test plan phase 1
  • Continued work on existing development with Percy  
  • Running long term tests to determine stability and categories for visual differences
  • Investigating visual shifting issue 
  • Found issues: #42028 and #41694