Why do customers choose Elastic for logs?

When your SREs and developers are trying to root cause analyze (RCA) an issue, it’s all about mean time to resolution (MTTR): search speed coupled with an intuitive query language can make the difference between a significant outage and a minor service disruption.

When you are evaluating logging vendors, read the fine print: while many tools may theoretically meet your requirements for search, their strict “schema-on-read” approach (wherein the data is not indexed) can make queries frustratingly slow for end users, particularly in a time of need (e.g., trying to root cause a production-affecting issue). 

In contrast, Elastic natively supports schema-on-write, wherein the data is indexed upon ingest in a manner optimized for search. This approach makes text searches blazingly fast by comparison to schema-on-read, significantly increasing the efficiency of developers and SREs who need to work with log signals on a daily basis. Further, by lowering friction, developers and SREs are more likely to adopt observability tools as a daily driver in their workflow, leading to better software and system quality.

Elastic also readily acknowledges the value of schema-on-read for use cases when you need to search your data in new ways. Runtime fields let operators extract new insights from existing data without reindexing. When performing RCA of a newfound issue, Elastic’s new ES|QL language enables analysts to rapidly build tactical queries by stringing together searches, data extraction, and enrichment in an intuitive, easy-to-read format.

As customers shift workloads to the cloud and Kubernetes, another practical constraint is introduced: the non-linear scale enabled by these technologies makes trivial log observation and alerting (e.g., searching for “*error*” in your log lines) arguably impractical.

Elastic recognizes this inflection point in the industry and has responded with a set of unique and best-in-class AI-powered tools to let SREs and developers offload the complexities introduced by scale. Our out-of-the-box machine learning jobs can alert when a spike or dip in log rate is truly an anomaly versus simply an expected fluctuation. This added intelligence reduces alert fatigue and, as a result, brings greater attention to real problems. 

Once an alert has been triggered, Elastic’s AI-powered log spike analysis tool can quickly and automatically assess why the log rate changed: Is it one application? Is it happening only in one region? Is there commonality in the log messages that comprise this change? Historically to answer these questions, SREs would need to invest hours manually comparing logs and metadata in an effort to determine if the change in log rate is a real problem, and if so, its root cause. With Elastic’s log spike analysis tool, this arduous process is reduced to mere seconds.

The noisy and bursty nature of log signals can quickly lead to runaway costs with competitive logging solutions.

For logging services native to cloud providers, cost structures are typically quite complex. This makes it challenging to understand how costs could be reduced. Further, charges related to cloud provider logging solutions often get obscured as a line item on an overall cloud provider bill, making them easy to miss, despite their substantial cost. By comparison, Elastic’s pricing is transparent and straightforward: we charge you for the storage and compute required to land, search, and retain your logs over a specified period of time. 

For other cloud-hosted logging solutions, high retention costs negatively encourage shorter data retention periods or offline data lakes, leading to less visibility of problems and operational status. Elastic’s frozen data tier keeps all of your older log data online and searchable while keeping costs relatively flat. You gain all the cost benefits of a data lake with none of the operational overhead associated with data rehydration.

Additionally, Elastic’s flexible deployment patterns give you the tools to keep logging costs within reason:

• To help reduce the cost of egressing logs from your applications running in the Cloud, we support VPC peering on all major cloud providers between your Cloud’s application VPC and the Elastic Cloud VPC.

• To help reduce the cost of test, development, and non-production environments, Elastic Cloud lets you quickly deploy, scale, test, and tear down environments; you pay only for the hours that you need.

Elastic strongly believes that the future of observability is one in which logging, APM, infrastructure monitoring, security, and profiling are fully integrated, providing a holistic view through a single unified interface. Doing so puts all your observability data in one place, free of friction. This in turn encourages your SREs to look at problems from multiple angles, helping them drive to a definitive RCA faster and with more accuracy.

Elastic Agent along with our support for OpenTelemetry makes it easy to ingest all of your observability signals into Elastic. Your log lines will be tagged with the same metadata as your APM data, allowing analysts to quickly and easily pivot between signals without requiring manual timespan or address correlation.

As your internal technology roadmaps lead toward tool consolidation and holistic monitoring, give consideration to which logging vendors are capable of ultimately ingesting all of your telemetry signals. Elastic allows you to start with logs, and over time add APM, infrastructure monitoring, profiling, and security to the same observability platform. Change is hard, but the cost and operational gains of a converged observability solution are immeasurable.

Finally, Elastic is also uniquely positioned to ingest, process, and tie together your business metrics and KPIs alongside your observability data. Do you know how your infrastructure impacts revenue? Or how a given outage impacted customer experience? This level of correlation, enabled by Elastic’s open data platform, empowers you to spend your infrastructure dollars in a manner optimized for your business; it can also help justify your spend on observability.

Elastic understands that you want to keep your operational staff focused on running your business applications and not running your logging infrastructure. To that end, Elastic continues to make big investments in simplifying the onboarding and operation of our observability platform.

Our ongoing commitment to OpenTelemetry allows you to invest once in fully instrumenting your applications without worrying about vendor lock-in. In April 2023, Elastic contributed our Elastic Common Schema (ECS) to Open Telemetry, further making Elastic the premiere platform for observing OpenTelemetry-enabled applications. 

Our fully fleet-managed Elastic Agent lets you deploy one unified agent per host with centralized configuration to easily configure and collect infrastructure and service telemetry. Elastic is also working with every major cloud provider to make sending logs to Elastic Cloud as easy as sending logs to a native cloud provider observability service.

We are changing the way in which the industry will think about onboarding log files. Our new experience will abstract away the complexities of indices, policies, and mappings, letting you focus on the data within the logs, rather than managing the log files themselves.

Are your developers and SREs coming from a competitive observability platform? Our Elastic AI Assistant can help onboard your end users, writing queries for them using natural language (i.e., “give me a list of pods whose memory was >= 90% of their limit”).

Finally, our forthcoming Elastic Cloud Serverless offering will provide observability (including logging) as a fully-managed Cloud service. With Serverless, Elastic will manage every aspect of compute, storage, and operations; you just need to bring the telemetry data and the end users!

In large part because of the innovations detailed here, Elastic has been recognized by Gartner and Forrester as a top 3 observability vendor. It’s also why big companies continue to turn to Elastic to meet their logging needs. But don’t take our word for it; see for yourself how Elastic is bringing new value to the venerable log line. You can play with our demo cluster here, or test against your own log data with a free Elastic Cloud trial.