Emerging trends in observability: GAI, AIOps, tools consolidation, and OpenTelemetry

As technology evolution continues at its rapid pace, so does observability. Observability is becoming critical to driving positive business outcomes, and we wanted to understand how users are evaluating trends and their impact over the coming years. And while every organization approaches observability around its own unique needs, there are usually specific trends and shifts that the broader industry undergoes. 

Elastic® recently surveyed over 500 observability decision-makers around key topics in modern observability. Respondents were asked about the progress and benefits they are seeing in their observability initiatives, and the impact of generative AI and OpenTelemetry on observability. The survey, completed with Dimensional Research in January 2024, delivered some surprising results that revealed the extent of progress in observability and the adoption of newer technologies we might expect to see. Here is a brief summary of the results:

As organizations adopt observability, the focus continues to shift from simply observing systems and finding and fixing operational issues to understanding the business impact. The survey shows that an overwhelming majority of respondents are focused on the business impact of observability, and the inclusion of business observability as a critical requirement is exciting to see. This evolving approach will drive how organizations evaluate and integrate observability with their business processes while increasingly using data to drive their business decisions. 

Survey respondents reinforced the continued challenge of having multiple tools, with the majority indicating that between one and three tools is the right number. As organizations continue on their tool consolidation journeys, it was revealing to understand the capabilities they are looking for as they drive this consolidation. A high percentage of organizations are increasingly looking for the ability to correlate data, as well as to leverage machine learning and analytics capabilities coupled with AIOps to identify and solve problems faster. These capabilities will enable organizations to deliver improved outcomes with faster issue resolution, cost reduction, and improved collaboration between teams. 

Observability practices, while still evolving, are yielding significant benefits for companies, with 94% of stakeholders reporting measurable improvements in areas like SLA compliance, automation, cloud optimization, customer satisfaction, and reduced incident resolution times. Specific benefits vary by role: IT and DevOps/SRE roles are experiencing faster deployment capabilities, while executives and managers are observing better responses to customer issues. 

The research highlights that the level of observability maturity correlates with the extent of benefits gained; early adopters see advantages, but those with mature practices report greater improvements. Early maturity stages are focused on enhanced response to customer issues, while the more mature stages see improved application deployment speeds. To further advance their observability capabilities, organizations are forming Centers of Excellence to centralize expertise, encourage collaboration, and standardize practices, with most respondents seeing the potential benefits of this approach.

Although there have been various levels of disillusionment with AIOps-related capabilities due to inaccurate or immature machine learning models that often worsened the problem with increased alert noise, the rapid development and evolution of generative AI is offering organizations the ability to empower users to derive faster insights across all levels of users in an organization. Generative AI capabilities paired with retrieval augmented generation (RAG) capabilities allow organizations to leverage the power of LLMs and private data (e.g., runbooks, alerts, business data) to deliver relevant and meaningful results and identify and solve problems faster while reducing noise. 

OpenTelemetry (OTel) continues to be the standard that organizations are increasingly converging on. There is a high level of interest in OTel in the survey with respondents indicating a broad range of adoption, from evaluation and experimentation to production deployment. While OTel is still in its early stages, the promise of OTel and the high level of organizations evaluating it continues to establish it as the de facto industry standard for observability. 

The survey also highlights the challenges of OTel adoption, ranging from concerns over technical support, lack of vendor support for OTel signals, and a need to wait for increased market adoption. As the community and standards continue to converge and evolve, we expect a majority of customers to want to adopt OTel, making broader vendor support and easier integration with OTel a critical requirement when choosing platforms.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.