News

Elasticsearch Service: Data Transfer and Snapshot Storage Pricing

By
Yoav Derazon

Back in August 2018, we introduced a significant price reduction to Elasticsearch Service pricing, along with a whole slew of new features like deployment templates, new instance types, and hot-warm architecture support. In the same pricing announcement, we noted that we would start breaking out and charging for snapshot storage and data transfer costs to give you more control over your usage and spending. During the promotional period, these new charges were fully discounted, i.e. they appeared as zero cost line items in your monthly invoices.

The promotional period on the data transfer and snapshot storage costs will be ending on February 28, 2019. This blog post describes these charges in detail, and provides some tips and advice on how users can control these costs.

Snapshot storage pricing

Snapshot storage costs are tied to the cost of storing the backup snapshots in the underlying IaaS object store (e.g. S3 on AWS or GCS on Google cloud). These storage costs are NOT for the disk storage that persists the Elasticsearch indices, as these are already included in the cost of the cluster (more generically called the “deployment” these days).

As is common with all cloud providers, we meter and bill snapshot storage using two dimensions:

  • Storage size (GB/Month)
  • Storage API requests (1K Requests/Month)

Storage size is calculated by metering the storage space (GBs) occupied by all the snapshots of all the deployments tied to an account. The same unit price applies to all regions. To calculate the due charges, we meter the amount of storage on an hourly basis and produce an average size (in GB) for a given month. The average amount is then used to bill the account for the GB/Mo used within a billing cycle (a calendar month).

For example, if the storage used in April 2019 was 100GB for 10 days, and then 130GB for the remaining 20 days of the month, the average storage would be 120 GB/Mo, calculated as (100*10 + 130*20)/30.

Storage API call costs are calculated by counting the total number of calls to backup or restore snapshots made by all deployments associated with an account. Unlike storage size, this dimension is cumulative, summed up across the billing cycle, and is billed at a price of 1,000 requests.

As of this writing (Feb 1, 2019), the rates for these two dimensions are:

  • Storage size - $0.033 per GB/Mo
  • Storage API Requests - $0.0018 per 1,000 API calls

We provide a free allowance of 100 GB/Mo to all accounts across all the account deployments. Any metered storage usage below that amount will not be billed. Whenever the 100 GB/Mo threshold is crossed, we bill for the storage used in excess of the 100GB/Mo free allowance.

We also provide a free allowance of 100,000 API requests to all accounts each month across all the account deployments. Once this threshold is passed, a charge will occur for the use of API requests in excess of the free allowance only.

Note: A single snapshot operation does not equal a single API call. There could be thousands of API calls associated with a single snapshot operation, as different files are written, deleted, modified etc. The price we list is in 1000's of API calls, meaning: $0.0018 for 1000 API calls or $1.8 for a million calls.

Data transfer pricing

Data transfer costs account for the volume of data (payload) going into, out of, and within the Elasticsearch deployments.

We are set to meter and bill data transfer using three dimensions:

  • Data in (free)
  • Data out
  • Data intra-deployment

Data in accounts for all of the traffic going into the cluster. That includes index requests with data payload, as well as queries sent to the cluster (although the byte size of the latter is typically much smaller).

Data out accounts for all of the traffic coming out of the cluster. That includes search results, as well as monitoring data sent out of the cluster. The same rate applies regardless of the destination of the data, being either to another region, to the internet, or to the same region but a different account.

Data intra-deployment accounts for all of the traffic sent between the components of the deployment. That includes mostly the data sync between nodes of a cluster, which is managed automatically by Elasticsearch cluster sharding. It also includes data related to search queries executed across multiple nodes of a cluster. Note that single node Elasticsearch clusters may also incur intra-cluster charges accounting for data exchanged with Kibana nodes or other nodes such as machine learning or APM. These charges are expected to be lower in such cases.

The data transfer usage is calculated similarly to the storage API requests, in that it is summed up to a cumulative amount within a billing cycle.

As of this writing (Feb 1, 2019), the rates for these three dimensions are:

  • Data in - $0 per GB/Transferred - This dimension is free
  • Data out - $0.032 per GB/Transferred
  • Data Intra-deployment - $0.016 per GB/Transferred

We also provide a free allowance of 100 GB/month, which covers data out and data intra-deployment separately, and across all the deployments of the account. Once this threshold is passed, a charge will apply for any data transfer used in excess of the 100GB/Month free allowance.

FAQ

Where can I see my snapshot storage and data transfer costs?

In order to make it easy to track the on-demand costs of snapshot storage and data transfer, we have added line items to invoices that are available for download in the user console. In addition, we will be adding month-to-date view of accrued costs in the user console to better estimate the next charge amount.

Example invoice:

When and how will this be rolled out?

The new charges will appear in your February 1, 2019 invoice (for the use in January 2019). However, the charges will be fully credited. This will serve as an estimate for your data transfer and storage charges in the following months. The same exercise will be repeated for the March 1, 2019 invoice (for the use in Feb 2019), and actual billable metering will begin March 1, 2019 to appear on the April 1, 2019 invoice.

How can I control my snapshot storage costs?

The way snapshots work in Elasticsearch is by saving data incrementally at each snapshot event. This means the effective snapshot size may be larger than the size of the current indices. The size gets bigger as more data is used in the cluster, as well as when the data is changed frequently (added/deleted/modified records). In order to allow control beyond adapting the changes to the data (which is not always practical), we included an advanced parameter in the Elastic Cloud user console under the snapshots sub menu, called Snapshot count. We have kept the current default of 100 snapshots (rolled over), but this can be changed to any value between [2 and 100].

Warning: Reducing the number of snapshots effectively reduces the retention period of indices. This means that only a recent restore point will exist and may expire quickly.

As for API requests, these are executed every time a snapshot is taken or restored. While restore is not typically a frequent option, snapshots are taken by default every 30 mins to maintain a recent and fresh restore point. We have included a new parameter named Snapshot interval, which can be changed up to 24 hours, resulting in less API calls.

Warning: Reducing the snapshots interval may result in partial data loss, as a restore operation of an older snapshot will not account for all the data that has changed since the last snapshot.

Finally, in case of any implemented logic using Elasticsearch API to create/restore snapshots, it is advisable to revisit that process to avoid excess charges.

How can I control my data transfer costs?

Data transfer out of deployments and between nodes of the cluster is harder to control, as it is a function of the use case employed for the cluster and cannot always be tuned. Some cases of batch queries executed at frequent intervals may be revisited, if applicable.

How will this affect my bill?

Given the varied ways of using Elasticsearch in different use cases, it is hard to predict the exact snapshot storage and data transfer costs for your account. For this we implemented two “grace” bills that will show the amount that would have been charged, but zeroed out the charges. We hope that this helps to better estimate your costs, and prepare for the change.

I’m an Annual Gold/Platinum customer. Will this affect my bills as well?

Existing annual customers under contract, including Standard Annual, Gold, and Platinum tiers will not have any additional costs until they renew their contract under the new terms, in effect since Jan 1, 2019. All new annual customers will be billed using the new pricing, same as new monthly customers.

My deployments are in GCP. Will this apply to me as well?

Yes. Snapshot storage and data transfer costs apply to all cloud providers and have the same rate, as of the time of this blog post.