Elastic data tiering strategy: Optimizing for a resilient and efficient implementation

Organizations love Elastic for its speed, scalability, customizability, and functionality. Because of this, they often find new use cases for Elastic. This becomes a challenge when large amounts of data are ingested without considering how the data is stored, managed, and used, which can lead to bottlenecks in data management. As data grows, their current setups struggle to handle new needs, reaching the limits of their hardware and licenses.

If your organization is experiencing these issues, the solution is more manageable than you might expect.

The way to overcome this challenge is by defining a data strategy that aligns with your business objectives. Instead of collecting and retaining data based on arbitrary requirements, ask yourself the following questions:

1. What data needs to be collected to drive a business objective?

2. How often is this data used?

3. Is there an expiry date after which this data is no longer valuable?

4. Are there compliance requirements for this data?

Based on the answers to the questions above, organizations can create a business-driven data strategy to optimize the way that data is being stored and utilized, maximizing their existing investment in Elastic.

To showcase the benefits of adopting this strategy, let’s explore a case study of a customer who has undergone this process. 

This customer typically processes 5TB of data per day and handles an average of 250,000 events per second. However, at times the volume increases to 7TB per day and 350,000 events per second. The Elastic implementation for this customer was focused on ingesting a high volume of security data and making it available to the security operations center (SOC) team to search for information on cyber incidents and fraud investigations.

This implementation was so successful that the customer added new use cases that required longer data retention and faster search capabilities from a wider range of data sources. They targeted the following business outcomes:

• Log optimization: By optimizing their data tiers, organizations can enhance their log management practices, ensuring that they retain the right logs for the right amount of time, improving operational efficiency, and compliance adherence.

• Improved license utilization: Efficient storage tiering means better license utilization, allowing organizations to make the most out of their existing resources and potentially avoid unnecessary licensing costs.

• Enhanced business efficiency: The ability to find insights from logs more efficiently can lead to improved business efficiency, enabling faster decision-making and more informed strategic planning.

• Onboarding new use cases: With optimized data tiers, organizations can easily onboard new use cases, expanding their data analytics capabilities without significant infrastructure investments.

• Clear data strategy: Optimized data tiering contributes to a clear data strategy, ensuring that data is reliable, easily accessible, and governed effectively, laying the foundation for data-driven decision-making.

Data tiering is a complex and nuanced topic and it deserves its own blog to unpack. However, for the purposes of defining a data strategy, the different data tiers can be simplified into three main uses: ingest, search, and store.

• Ingest (hot tier): Ingest data as quickly as possible with minimal latency.

• Search (hot and warm tier): Search data quickly and crunch large data sets.

• Store (cold and frozen tier): Store data for as long as required and perform low-frequency ad-hoc searches.

Understanding the range of data retention requirements is crucial for compliance and efficient data management. Different regulations necessitate varying retention periods:

• SOX retention requirements: 7 Years

• HIPAA data retention requirements: 6 Years

• PCI DDS data retention requirements: 1 Year

• Basel II data retention requirements: 3–7 Years

• GDPR employee records:

  • Wages: 3 Years

  • Tax records: 6 Years

  • Name, address: 3 Years

• Fair Labor Standards Act: 2–3 Years

Previous architecture
The previous architecture had two data centers with a four-tier storage implementation catering to various data handling needs. This implementation required more hardware, licenses, and overhead in operational management.

The customer retained all logs for 90 days, regardless of how the data was used.

• 7 days hot

• 2 days warm

• 10 days cold

• Remaining days in frozen

The customer had the same hardware in both the hot and warm tiers. The warm tier was used purely for force merging indices for searchable snapshots. The warm and cold tiers were very underutilized in both CPU and storage. The frozen tier was narrow, resulting in slow historical searches.

A tiered data strategy allows organizations to optimize performance for recent data while efficiently storing large volumes of data. By leveraging shard allocation awareness, organizations can define the characteristics of each tier and schedule the migration of indices according to the data strategy. This ensures that data is stored on the most appropriate hardware tier at any given time, balancing performance and cost considerations.

Memory to storage ratio is a crucial consideration when planning Elastic growth. Below are the four storage tiers available to Elastic customers:

• Hot tier: Optimized for ingestion and search performance, typically using high-speed SSDs with a memory to storage ratio of around 1:30

• Warm tier: Optimized for storage capacity, utilizing SSD or HDDs with a memory to storage ratio of approximately 1:160

• Cold tier: Optimized for storage capacity using a searchable snapshot as a replica (Although the storage ratio is the same as the warm tier, the removal of a local replica halves the storage requirements.) 

• Frozen tier: Optimized for archival purposes, employing cheap snapshot storage with local disk cache to provide a memory-to-storage ratio exceeding 1:1,000

In our analysis, we assessed the total cost of ownership (TCO) for various storage configurations to optimize another customer's Elastic implementation. Below is a detailed breakdown of these configurations and their associated costs:

•  Self-managed ES cluster

•  1TB daily ingestion

•  Total retention 365 days

Optimizing storage tiering is not just about saving costs; it's about enabling organizations to evolve and adapt to new challenges and opportunities. 

By approaching platform optimization challenges using the data strategy principles, organizations can facilitate new use cases, improve data reliability, and enhance their overall data strategy. Check out our documentation to learn how your organization can build a resilient and efficient implementation of Elastic using data tiering.