Kibana, Aggregation Execution Order, and You
By now you may have spotted those crafty little arrow buttons in the Visualize screen of Kibana 4 and said “Hey, what are you doing over there? And what are you up to?". Well, those buttons control the aggregation execution order. This concept defines how Elasticsearch goes about analyzing your data, and how Kibana displays the result.
Let's define a common scenario: Find me the most active users over time. Easy enough right? Well yes, but your demand is ambiguous and your goals are unclear. What defines the “most active users"? Let's set more parameters: One year, by week, top 5 users. Now we're getting closer, but there are two ways we can interpret all that:
- The top 5 users for the year, and their activity each week
- Each week's top 5 users, for a year
Top 5 users in each week, for a year
In this screenshot, we run the date histogram first, followed by asking for the top 5 users. This creates a bucket for each week of the year. Within each of those weeks, we find the top 5 users. Because that top 5 could be, and is in this case, different from week-to-week, you see far more than 5 users in the legend.
Further, if we look at the aggregation request in the shaded section, we can see the date histogram is requested first, with the terms aggregation within the date histogram. The result is that we see weeks in which some user has an outsized activity, even if they haven't been active in any other portion of the year. This lets us spot outliers in any given week.
Top 5 users for the year, and their weekly activity
Now, we click the up arrow to move the terms aggregation above the date histogram. We now calculate the top 5 users over the entire year, then create a date histogram for each. This results in just 5 legend values. However, we now see the users that are consistently very active instead of the spikey outliers.
Go forth and aggregate
So there you go: those arrows matter. Aggregation execution order applies to almost every chart in Kibana and significantly influences both what you see on the chart, as well as the conclusions you can draw from the data.
And finally, if you think you have a good Kibana success story, we'd love to hear it. Give us a shout at [email protected] or on Twitter and we'll help share your successes with the world!