What’s the difference? Elastic data tiers and Amazon OpenSearch Service tiers

The concept of data tiers is present in most data platforms, especially in those that deal with observability and/or security tools. The volume of data collected by these tools is usually very high, with thousands/millions of events per second being processed and made available for searching, dashboarding, and alerting. Observability and security also have a shared characteristic: the most recent data is also the most valuable, as teams administering these tools rely on the signals being collected to take immediate action in case of problems.

So it makes sense for data to be ingested and stored with the fastest possible hardware and moved “down” to cheaper, less powerful hardware as time passes.

Elastic has five tiers, which can be independently or collectively utilized depending on your specific use case:

• Hot: Your data always arrives here first, and it’s highly-available in real time, scalable, and offers the best possible performance (assuming best practices are adhered to). This is where you keep data that you need to access and manipulate frequently.

• Warm: This tier allows for more cost-effective hardware utilization, where data that isn’t in immediate demand (but still relatively important) can reside. You can move data to this tier and optimize it (by force-merging the segments for instance), so searches are as fast as possible. Data in this tier is still scalable with replicas, just like in the Hot tier so as to meet search demands if needed.

• Cold: Here it ensures at least one copy of the data will always be allocated to the node and searchable at any given moment. The Cold tier uses object storage to aid in data restoration should there be a failure or a need to alter the topology of the cluster.

• Frozen: In this tier, data is less frequently accessed and allows for cost savings as it leverages the lowest cost storage and reduces compute resources. Data is searchable, but it must be restored back into a searchable state, which is done automatically and transparently with Elasticsearch®’s searchable snapshots. 

• Snapshots: Snapshots are essentially data backups — point-in-time copies of your indices. These can be used for various purposes, such as data recovery in case of a loss, creating clones of indices for testing or staging environments, or for migrating data between clusters. Snapshots are stored in a repository, which could be on different storage systems like a local filesystem or object storage (e.g., GCS, S3) and must be manually restored for data to be searchable.

Now we can compare the tiers in terms of data access capabilities: can data be read and written or is it read-only? Does it need to be manually restored or is the “thawing” process automatic? Here’s what each “band” represents:

Another important aspect to consider is the instance types used in each tier. It’s also important to note Elastic Cloud supports three major cloud providers (AWS, Google Cloud, and Microsoft Azure) with different hardware profiles on each. Amazon OpenSearch Service’s approach designates specific instances (like OR1 and Im4gn) to its service with particular software version requirements and restrictions on EBS volume support.

Both Amazon OpenSearch Service and Elastic Cloud on AWS utilize Graviton2 based instances, indicating a shared preference for the performance gains and cost efficiencies of AWS’s ARM-based chipsets. Elastic Cloud on AWS is less prescriptive about the exact use-cases for its instances, providing a selection that includes high compute with fast storage (Graviton2 instances) and a variety of more traditional choices (like C5d, M5d, etc.).

Naming conventions can be misleading, causing understandable confusion when trying to align business needs to data storage options among providers. Having a grasp on the actual capabilities of these tiers can help you make more informed and cost-effective decisions regarding data management.

This breakdown is meant to dispel misconceptions brought about by the naming overlap in data tiers between Elastic and Amazon OpenSearch Service. With this description of data tiers, you’ll be better positioned to organize your data strategically for performance and cost benefits. It’s critical to move beyond the names and understand the underlying mechanics of each tier to ensure your data strategy is both robust and efficient.

Please also see the following studies: Elasticsearch surpasses OpenSearch in cost efficiency along with how Elasticsearch outperforms OpenSearch while using fewer resources.