Welcome Prelert to the Elastic Team

I am happy to announce that Prelert and Elastic are joining forces. Ever since we started Elastic, our goal has been to allow users to easily find relevant data or insights within large amounts of data. Search is a wonderful way to do it, and the ability to slice, dice, and aggregate the data in an unconstrained way allowed users to feel they are in control of the data, compared to the other way around.

But we can take it a step forward, and with Prelert, we just did. Prelert has developed an unsupervised machine learning engine that can plow through large amounts of data and automatically find those insights our users today have been proactively finding using search.

We view the Prelert technology as a generic engine that can apply to many different use cases, which maps very nicely with what we are trying to do with the Elastic Stack. It has been proven to be extremely successful within specific use cases. Finding anomalies within transactions / operational metrics, detecting uncharacteristic user behavior, finding a population of attacking IP addresses, and much more.

Let me stop here and let Steve Dodson, Prelert CTO and Founder, share his thoughts:

At Prelert we work on developing machine learning technologies that allow users to understand the behavior of their data. Joining Elastic gives us a fantastic opportunity to bring this technology to a large range of users and data, and will allow us to execute on the vision we set out for Prelert over 7 years ago.

The journey to this point initially started when I was offered the opportunity to help diagnose an application issue on a large trading platform in an investment bank. Sporadically, traders were becoming disconnected from the trading platform and the goal was to proactively preempt this issue and diagnose the root cause. The problem was that the data was overwhelming, with 100s of log files from 100s of servers, and 10,000s of performance metrics from the applications and systems.

Simply searching through the logs or metrics didn’t yield sufficient insight into the problem, and it started to become clear that hidden in this data was the behavior of the system, and if we could model the normal behavior of the system we could isolate the unusual activity. Based on this premise we built out a prototype that statistically modelled the behaviour of the system via the logs and metrics allowing the customer to identify, diagnose and resolve the issue.

(First Prelert UI showing anomalies and root cause!)

The great thing was that at the same time we were developing this technology, users with large IT systems were moving from legacy monitoring solutions, to solutions that could collect and centralize storage of all their logs, performance metrics and data. Elasticsearch is a great example of this, and showed how this new generation of search technologies could be used by customers to give significant insight into their data. Also, the questions users were asking from this data were becoming broader, and instead of relying on silo’ed tools, users were using these platforms to answer IT Operations, Security, Business and other questions.

Layering Prelert on top of these technologies was the natural next step, as users were asking questions that required statistical and machine learning analysis of the data. Another really exciting discovery was that customers then started using Prelert to answer questions such as “which bus routes are congested?”, “is the temperature across my environment normal?”, “has there been a change in the number of retail transactions?” along with traditional IT Operations questions. This showed that our technology was not only robust to diverse use cases, but had the opportunity to be broadly applicable across diverse time series data.

(Current Prelert UI in Kibana)

However, a challenge we kept running into was that customers wanted our analytics more and more tightly integrated into the data platform. The really exciting part is that Elastic shared our vision of extending these platforms beyond search using machine learning, and so joining Elastic is the natural next chapter for Prelert, and it’s a privilege to be working with Steven, Shay and team.

Thanks Steve :). Let me cover the near-term future when it comes to the products.

Prelert has already worked on a wonderful integration with Elasticsearch and Kibana. It is used in 2 products, the API Engine, and the Integration for Elastic Stack. Now that we joined forces, we think we can improve this to be a much more integrated product, where Prelert becomes a feature of our stack. Think Prelert node(s). Within this context, we are going to put a Beta label on the existing 2 products, and work towards the more native integration towards a GA.

Prelert also developed a Splunk App, and we are announcing today the End of Life for the app. In the short term, nothing changes for Prelert customers but this announcement starts the timing for the maintenance period of this app. Importantly, the app will still be supported and all existing customer commitments will be upheld. We hope that any existing Splunk customers who are interested in using the Elastic Stack will find significant improvements in the user experience with our native Prelert and Elastic Stack integration. We will obviously be there for the existing customers to help migrate and transition to use Elastic products.

We are very excited about this, and we think this will prove to be a valuable addition to the Elastic user base, and a great, fully integrated, and improved product for Prelert’s customers.

Oh, and if you want to see Prelert in action and hear more about our developments, come join us at one of our Elastic{ON} Tour stops in the US, EMEA, or in Asia Pacific.