On Indexing Heartbeats and Ballet into Elasticsearch

Nobody expects a ballerina to open a tech conference keynote.

But that's just what we did. On March 7, 2017, we opened Elastic{ON} with a ballet performance shipping biometric and audio data into the Elastic Stack for visualization. This is a look at the effort that went into making it happen.

An Unconventional Idea

A little over a year ago, we pitched an idea. It went a something like this:

You walk up to the registration desk at Elastic{ON}. You grab your badge, it jangles familiarly as you loop it around your neck. You walk through the venue, taking it all in as you head to the main stage for the keynote.

Scanning the room, you spot a good seat and settle in. The lights start to flicker, indicating the presumed start to the conference with the keynote speakers mic'd, enthusiastic, and ready with a heartfelt hello and welcome.

Instead, the lights go out and the room goes quiet. A spotlight shines on a lone ballerina center stage.

She's wearing a simple ballerina getup, but there's also circuit-like objects attached to her. One of them starts to glow, Kibana dashboards appear behind her, and a familiar EKG line appears. It's her heart rate.

And then she begins to dance...

The brief continued on to describe other types of data capture and visualization using a variety of sensors related to a ballet performance. The idea was to demonstrate a creative and unexpected use of the Elastic Stack that pushed the boundaries of what people expect.

We casually circulated the concept to our peers to get their reactions. Dig it? Don't get it? Not true to Elastic? The majority response was positive. Eventually it made its way to Shay, creator of Elasticsearch and CTO. He had two comments:

• Will the technology be ready for it?
• Can you do it in a way so that a viewer of the performance could easily replicate it?

What we didn't hear was, "No." So we got to work.

The Proto-Ballerina

In a way, what became known as the Ballerina Project was made possible because of a popsicle, a hairdryer, and a warm Saturday afternoon.

We teamed up with a local Elastic community member we knew who had the right kind of software, hardware, and design chops to pull something like this off. Presented with the idea, he worked out a few ways to explore the viability of the project. One of them was by building an Arduino-based home weather station, using inexpensive parts bought off the internet and the Elastic Stack.

Only, the weather in Oakland, California in June isn't spectacularly that interesting. So he improvised and pushed the temperature readings to extremes with a popsicle and a hairdryer, resulting in a Kibana line graph that roller-coastered hot and cold faster than most of us had seen before.

While a temperature sensor the size of a cell phone built in a garage on a weekend was a long way from a wired-up ballerina, we learned that it was possible.

All that was left was to do it.

Dancing in a Crowd of "Data Thingies"

The industry of "data thingies" is vast and expanding. A multitude of products offer actionable insights and worthwhile value extraction from the growing oceans of data. Technology promises to help us to find meaning in data by connecting the dots to make the best decisions possible.

The trouble is, saying you work in "data" today is on par with saying you worked in "computers" a few decades ago: people tune out, eyes glaze over. Often, it's interpreted as you do something complicated that only some people will understand.

To an extent, this is true. Every collision event at the Large Hadron Collider results in one million bytes (1 MB) of data — there are about 600 million events per second. That's a lot of data. Most of us wouldn't know how to start sifting through it to find something meaningful. So, perhaps we do leave that to the particle physicists to sort out.

But there are many instances in which you don't have to be an expert. Every 140-character post to the Twittersphere is a data point — each is a record or a log of something that happened at a point in time. Looking across many of them will start to reveal a story, a trend, a pattern, or a series of unexpected connections. The same holds true for hundreds of failed Facebook login attempts (you're being hacked), or monthly toothpaste purchases from Amazon (you fear cavities), or electricity consumption in our homes (you care about energy efficiency).

Indexing heartbeats or g-forces during a ballerina's pirouette isn't all that different from analyzing electron collisions, monitoring disk space, or tracking CPU usage.

Everything is a data point. And the Elastic Stack gives you an incredibly powerful and flexible platform to explore data and find the stories within it.

The hardware embedded into gloves to track hand acceleration and elastic wrap to capture foot acceleration with the ballet shoes. Photo credit: Justin Oliphant.

Elastic, the company, was founded by those who are at their core inquisitive, inventive, and curious. Our users are no different. They tinker, play, fiddle, test, break, and build. They eat interesting problems for breakfast. Elastic technology allows them to do just that in a way that meets their needs.

One of our employees described Elastic as having a knack for solving a variety of problems well. "And it's not because we've told people we do it well," he said. "It's because they've found out for themselves. We've given this really powerful technology to the doers."

"And it's not because we've told people we do it well. It's because they've found out for themselves. We've given this really powerful technology to the doers."

This concept was core to the spirit of the Ballerina Project. It should serve as an invitation to anyone exposed to it to be curious about the world around them and start exploring it. Because it's littered with clues just waiting to be pieced together to reveal something interesting.

Wondering what would happen if you wired up a ballerina with sensors and visualized her movements? Go find out. You shouldn't be limited by what technology will allow you to do. That's the power of the Elastic Stack.

This is Nuts. Let's Do It.

While we have a propensity to embrace the risk that goes along with live demos during Elastic{ON}, the ballerina was pushing it to a new level of gutsy.

But everyone was on board — they just got it. The story made sense. The concept fit with who we are as a company. It colored outside the lines, pushed the boundaries of what was possible, and promised to resonate with the audience in a creative and relevant way. And that's what made it compelling and worth it.

We set about building out a team to make it all happen. It included folks from a variety of backgrounds: engineering, product management, a ballerina, costume design, choreography, music composition, production, design, stage direction, and more. There were many disciplines brought together to bring the performance to life.

We set up a cadence of rehearsals every two weeks for three months. Over that time we proved out various parts of the plan: how to integrate rigid computer chips, battery packs, and wiring into a stretchy, thin leotard that wouldn't hinder the ballerina's movements; how to choreograph a dance that would, in turn, make the charts dance and be understandable to the audience; and how to harden the simple architecture to ship data from the wifi-enabled sensors to Elasticsearch and sharpen the visualizations in Kibana.

We had our work cut out for us. But it's been said that things worth doing seldom come easy.

The ballerina testing out a glove prototype and viewing the real-time results in Kibana during a rehearsal in February 2017. 

Show, Don't Tell

We thought a lot about what to show and how to show it. Biometrics can be messy.

In the end, we landed on four types of metrics with three types of visualizations: heart rate (voltage changes via EKG) with a line chart, acceleration (hands, feet, and hips) with a line chart, location (x, y movement via infrared light) with a heatmap, and sound (frequency) with a bubble chart.

For how to show it, we wanted to retain the problem-solving element. We did not want to take extra steps to label, spell out, or narrate what was happening on stage. We wanted to let the data, visualizations, and dance do the talking and, in turn, let the audience make the connection, solve the puzzle. That is the nature of our audience, after all.

Perhaps the hardest part of the performance was finding ways to make sure the audience could understand the visualization they were seeing and connect it to what was happening on stage. (Seeing hand acceleration visualized simultaneously across the x, y, z axes is overwhelming and chaotic.) Solving this meant approaching the problem from two main angles: choreography and the software.

Short bursts and prolonged pauses were our allies. When isolated to a particular limb or movement, these busts and pauses served as choreographed teaching moments for the audience. Onlookers would see a specific movement (or lack thereof) correspond to a Kibana chart showing a flat line that would then arc or curve and then flatten out again.

For example, one of our favorites was what became known as "the foot scribbles" or frappés, in ballerina lingo. They were incredibly effective at showing a connection between the dancer and the graphs. When she frappéd her right foot, the top line activated in Kibana while the bottom remained steady. When she switched sides and frappéd her left foot, the bottom line activated and the top remained steady.

The ballerina practices her frappés (or "foot scribbles). Her right foot is visualized on the top line chart and her left foot appears on the bottom line chart. 

While having those moments sprinkled throughout the performance was key, we wanted to preserve the natural beauty of the ballet itself. We just needed to give the audience a chance to understand what was happening so they could enjoy what would follow next.

The rest of the work fell to the Elastic Stack. How could we refine and customize the visualizations to effectively communicate with the audience? 

The key factor here was quick iteration. We could plow through a list of questions and test ideas and concepts in minutes. What if we only show two acceleration vectors instead of three? What does it look like with one? What if it's an area chart instead of a line chart? What if we applied a 95th percentile aggregation to it? Does the visualization make more sense if we show a 15-second timeframe rather than a 5-second window? Does the line look better if it's blue rather than purple?

This allowed us to eliminate and fine-tune our options to zero in on a best possible option for the performance. Throughout, it was empowering to really make the visualizations our own, to view the data the way we wanted.

(Note that there is lots more to say about how each visualization and sensor set worked. We're saving those details for future blog posts.)

The Ballerina Danced, the Graphs Danced

Even with final choreography, visualizations, hardware, and costume, there will still be unknowns to contend with in the days leading up to the final performance on March 7, 2017. The Elastic{ON} stage environment was totally new — it was constructed less than 24 hours before we started our final dress rehearsals. From wifi gremlins to projection clarity onto massive screens, we had a handful of hours to address and resolve any outstanding issues. (In fact, we had to swap out the heatmap section because of too much infrared interference from the stage lighting.)

Regardless of what wrenches could be thrown into our spokes, as long as the software conveyed an accurate representation of what was happening in real life the performance would be a success.

This meant that if the ballerina unexpectedly tripped or a sensor died, as long as the Elastic Stack reflected that outcome, the technology was doing its job. Because knowing when you don't have data can be just as important as knowing when you do. Knowing when something isn't working can be just as important as knowing when it is.

In the end, it worked. The ballerina danced, the graphs danced, and as best we can tell, the audience understood it and appreciated it.

And hopefully they walked away with an inkling to think differently about their existing projects or sparked ideas for new ones. It certainly did for us — and we're excited to see where it leads us next.

Learn more about the nuts and bolts of the ballerina performance and see what else happened at Elastic{ON} 2017. 

Also, check out this blog post from the project's choreographer, sharing her perspective on the intersection of technology and the arts.

Wired up ballet slippers. Photo credit: Justin Oliphant.