Troubleshooting Kibana health

You can access the Elasticsearch® database through Elastic’s UI product: Kibana®. While most Elastic® services run from Elasticsearch, a handful of key features run from within Kibana’s servers: Telemetry, Rules (with associated Actions), Reporting, and Sessions. We’ll walk through troubleshooting overall Kibana health and its most popular plugins. This content is generally true for 7.x and 8.x, but it is tested directly against v8.7.1.

Once you have these responses stored respectively as kibana_status.json and kibana_task_manager_health.json, then you have the two core APIs Support inspects from a Kibana’s Support Diagnostic pull to check Kibana’s overall performance.

You can inspect the answers to our step-wise dependencies manually or use the third-party JQ tool to automate the JSON parsing. In this blog, I’ll share the JQ commands, but the language’s expressions are simple enough to read for you to manually check outputs if desired.

You can start by confirming if Kibana is currently available or degraded:

As needed, you can check Kibana logs if status health flapping is suspected after initial startup.

We’ll cover the Task Manager health in the next section to give it the space-time it deserves.

4) Plugins. To check all plugin health, you can run the following JQ:

As desired, you can filter to problematic plugins by appending | grep -vw ‘available’. 

If you show multiple plugins in degraded status, it might be worth running the following more targeted query. This query highlights the plugins most likely to cascade into errors for the other plugins or surface as errors to your users:

I’ve included reason in this latter query, which can return a quite verbose output to help show how plugin problems can root in other unhealthy plugins. Let’s look at an example output:

Looking at this root-up, we show the Task Manager is unhealthy, which causes the following plugins to be unhealthy: security, savedObjects, and reporting. Next-step or eventual cascade this causes ruleRegistry and reporting (again) to report degraded. So for this output, the only plugin to focus on resolving is the taskManager and then we can revisit once we confirm it’s not cascade impacting the other plugins.

I’ll note these plugins can have a potential impact on users’ experience (e.g., loading Discover — see our troubleshooting blog).

The Task Manager itself is a roll-up of four subsections, so let’s next check how its child objects are doing:

For most performance issues, these subsections will report obvious health issues via their status. You can confirm the subsection status’s roll-up via its own child data: capacity, config, runtime, workload.

So at this point, the story I have rolling in my head is that Kibana is experiencing duration performance issues due to Task Manager having capacity warnings. So from the capacity troubleshooting doc, I’ll confirm how many servers Kibana thinks it has versus proposes.

So, we have one observed server and Kibana estimates that it needs two servers. Kindly note some stray troubleshooting details:

• Observed is looking toward recent server IDs, which may rotate upon Kibana restart and may report higher than actual.
• Task Manager’s proposed servers stat is always looking at horizontal scaling; however, vertically scaling may also potentially answer — kindly confirm via our scaling guidance.

In this particular case, I wanted to dig more into why Kibana thought it needed more resources before scaling up. So cross-comparing Task Manager Health’s runtime (even though it wasn’t erring), I ended up determining I had pretty significant drift going on.

Since drift reports in milliseconds, this indicates my p50 drift is a bit over a minute. That’s pretty backed up. So I’m going to look at drift_by_type in order to see if Kibana features are backed up evenly across the board (which is very uncommon) or which features are backing up the others:

You’ll note I filtered .p99>80000, which was a figure determined by the output of my earlier drift query (and laziness to write out the full number instead of just adding 0s after the 8 to the right placeholder). 

This output tells us some one-off Features (e.g., Fleet-Usage-Logger and reports:monitor) are elevated, but I care less since those aren’t re-occurrent, user-impacting tasks. Instead, the alerting:siem.mlRule and alerting.siem.queryRule appear to be my drift inducers, maybe with a secondary impact by actions:webhook.

Kindly note, this targets system index data, which can be read by admin users but should never be directly updated via this backend view. As Support, I prefer to query this way because this index is Kibana space-agnostic versus the frontend APIs, which are space-aware. 

Expensive. For the sake of our example, let’s assume quantity wasn’t our problem. Instead, we’ll want to check if some amount of Rules are more expensive than the rest. In my experience, this is quite common (e.g., 1 Rule of 3,000 takes 1 minute but the rest take <7 seconds). 

In order to troubleshoot Expensive Rules, we have a couple of options: 

1. (Space-agnostic) use Kibana’s Event Log to run the Expensive Rule query 

2. (Alerting) use Stack Management > Rules “Duration” table

3. (Security) use Security > Rule Monitoring tab

Options 2 and 3 are very user friendly and I highly recommend them. Option 1 is Support’s go-to since we don’t get to see user UIs and screenshots aren’t the same. So much in fact that I frequently use a variation of Option 1 to distill the histogram response into a table of problematic Rules to investigate:

You’ll note now-24h can be expanded (e.g., now-7d to see longer/shorter historical time frames). 

The baseline duration expectations I try to set with users for Rules impacting overall performance are that: 

• 10 seconds every 1 minute may impact or may be OK depending on resources
• Anything longer than 1 minute is likely impacting
• It seems egregious to have to say but I run into it weekly: Rules taking >5 mins to run are definitely problematic

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.