Kibana under the hood: object persistence

Users of Kibana create saved searches, visualizations, dashboards, and other such objects. Kibana, of course, has to persist these objects somewhere so they can be loaded up next time users start it up. As Kibana connects to Elasticsearch for querying users' data anyway, it conveniently uses Elasticsearch to store its objects as well.

This blog post explores some of these objects, how they are stored in Elasticsearch, and when they are created. The goal is for this information to serve as a useful tool not only for debugging but also for administrators who might want to deploy Kibana in an automated, repeatable fashion today. Please note, however, that these objects are internal data structures that may evolve over time. Future versions of Kibana may provide REST APIs to programmatically — and safely — manipulate these objects.

The special index

Kibana stores its objects as documents in the .kibana index in Elasticsearch. The name of this index can be changed via the kibana.index configuration setting (starting with Kibana 4.2; prior to that this setting was named kibana_index).

Let's start at the beginning

When you start with a completely fresh install of Kibana there is, of course, no .kibana index in Elasticsearch yet. This index is created when you start the Kibana server. At this point the index contains two document types:

• config, which contains exactly one document. This document's _id is the version of Kibana that you are running (e.g. 5.0.0) and it has a field, buildNum, that contains the build number (e.g. 8467) of Kibana that you are running.
• server, which contains documents that Kibana uses to identify itself in tools such as X-Pack Monitoring.

Index patterns

When a user loads up Kibana in their browser for the first time, they are required to create an index pattern. Of course, they can add more index patterns at a later time as well.

Whenever users create a new index pattern, a document is created in the .kibana index, under the index-pattern document type. Depending on the choices the user makes when creating the index pattern, this document will have the following fields:

• title: The title of the index pattern, for example, "logstash-*". This also becomes the _id of the document.
• timeFieldName: If the indices matched by the index pattern contain time-based events, the name of the field to use when filtering using Kibana's time picker control.
• fields: A string containing a JSON-encoded array of fields in this index pattern. Also where scripted fields are stored.
• notExpandable: Boolean field, determines whether to search the index pattern literally (when set to true) or to expand it and query only those indices in the pattern that fall within the currently-selected time range (when set to false, which is the default). Applicable only when an index contains time-based events, that is, timeFieldName is set.
• intervalName: If the user specifies that index names are based on event times (deprecated, by the way), this field will store the interval at which the index names rotate, for example, "hours".
• fieldFormatMap: This field is defined in the mapping but is not populated when an index pattern is first created. We will cover this field in a later blog post.

Also, when users create the very first index pattern, it becomes the default index pattern for Kibana to use as well. This bit of information — specifically the _id of the index pattern document — is recorded in the defaultIndex field of the config type document introduced in the Let's start at the beginning section above.

Saved searches

After creating an index pattern, users will typically start on the Discover page. Here they will explore the data in their index pattern by searching through it, focussing on certain fields in the results, etc.

Sometimes users will want to save their search on the Discover page for later use. They might want to open it later on the Discover page or add it as a panel to a dashboard. When users save a search, a new document is created in the .kibana index under the search type. It contains the following fields:

• title: the name of the saved search specified by the user.
• description: a longer description of the saved search. Users cannot specify this when saving the search. To set this, users must go to the Objects tab on the Settings page and edit the appropriate object under the Searches tab. The description is shown when a user tries to open a saved search on the Discover page or add it as a panel to a dashboard.
• kibanaSavedObjectMeta: This is an object with one property:
• searchSourceJSON: A string containing a JSON-encoded object that forms the basis of the search request Kibana will send to Elasticsearch. It contains information such as the index pattern name and the query the user has typed into the search bar or the filters the user has applied.
• columns: an array of the fields to display in the search results. For index patterns containing time-based events, the timestamp field (see timeFieldName in the Index patterns section above) is always shown so it is not included in this array.
• sort: an array of two elements: the first one is the name of the field to sort the search results on, and the second one is the direction of the sort.
• hits: intended to track popularity of this saved search. Not yet implemented so currently always set to 0.
• version: the version for the schema of this persisted document. Reserved for future-proofing, in case the schema is changed. Currently always set to 1.

Saved visualizations

When users create a new visualization via the Visualization page, a new document is created in the .kibana index under the visualization type. It contains the following fields:

• title: the name of the saved visualization specified by the user.
• description: a longer description of the saved visualization. Users cannot specify this when saving the visualization. To set this, users must go to the Objects tab on the Settings page and edit the appropriate object under the Visualizations tab. The description is shown when a user tries to open a saved visualization on the Visualization page or add it as a panel to a dashboard.
• kibanaSavedObjectMeta: (Same as described in the Saved searches section above)
• visState: A string containing a JSON-encoded object that has properties for the type of visualization, its parameters (many of which are editable by the user via the Options tab in the visualization editor), and the aggregations that will be requested from Elasticsearch in order to render the resulting data.
• uiStateJSON: A string containing a JSON-encoded object that has properties for settings such as the colors used in the visualization, whether the spy mode panel is open and in what state, etc. This differs from visState in that visState is a serialization of properties of the Vis class in the code whereas uiStateJSON is a generic bag of properties that could be attached to visualizations or dashboards (as we’ll see below).
• version: the version for the schema of this persisted document. Reserved for future-proofing, in case the schema is changed. Currently always set to 1.

Saved dashboards

When users create a new dashboard via the Dashboard page, a new document is created in the .kibana index under the (surprise, surprise!) dashboard type. It contains the following fields:

• title: the name of the saved dashboard specified by the user.
• description: a longer description of the saved dashboard. Users cannot specify this when saving the dashboard. To set this, users must go to the Objects tab on the Settings page and edit the appropriate object under the Dashboards tab. The description is shown when a user tries to open a saved dashboard on the Dashboards page.
• kibanaSavedObjectMeta: (Same as described in the Saved searches section above)
• panelsJSON: A string containing a JSON-encoded array representing the panels in the dashboard. Each panel is a JSON-encoded object that contains information on the type of the panel (visualization or search), its ID, its size (size_x and size_y), and location (row and col) on the dashboard.
• optionsJSON: A string contain a JSON-encoded object for dashboard options such as whether to use the dark theme.
• uiStateJSON: A string containing a JSON-encoded object that records the UI states for each visualization panel in the dashboard, but only if they are different from the UI state of the corresponding saved visualization.
• timeRestore: A boolean flag to indicate whether to store the time range (at the time of saving the dashboard) with the dashboard so that next time the dashboard is opened, it uses the same time range.
• timeFrom: If timeRestore is set to true, the lower bound of the time range, e.g. "now-15m".
• timeTo: If timeRestore is set to true, the upper bound of the time range, e.g. "now".
• hits: intended to track popularity of this saved dashboard. Not yet implemented so currently always set to 0.
• version: the version for the schema of this persisted document. Reserved for future-proofing, in case the schema is changed. Currently always set to 1.

This post covered persistence for the fundamental objects in Kibana. In a future post we’ll cover more advanced objects and lifecycle operations.

As always, if you have questions about Kibana feel free to ask them in our discussion forums at https://discuss.elastic.co/c/kibana or chat live with the Kibana team and community in the #kibana channel on Freenode IRC.

Thanks to Rashid Khan and Spencer Alger for shedding light on the rationale behind some of the decisions that went into designing the structure of the documents persisted in the .kibana index.