Scaling Elasticsearch on a hyper-converged infrastructure at IHG
This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.
InterContinental Hotels Group, or IHG, is a multinational hospitality company that operates over 5,000 hotels in nearly 100 countries around the globe. IHG’s global technology organization, based in Atlanta, GA, has about 13,000 corporate employees, many of whom are using Elastic to do their jobs. With the success and continued adoption of IHG’s Elastic platform across the company, IHG has explored interesting ways of scaling its infrastructure, most notably the shift to hyper-converged. “We needed a cost-effective way to expand our ELK presence, which is what led us down the path to hyper-converged,” says IHG Systems Engineer Benjie Godfrey.
Hyper-converged infrastructure is an IT framework that aims to increase scalability by combining compute, network, and storage functions in a single box. Converged infrastructure, in contrast, packages compute, network, and storage together as discrete elements within a validated, standardized platform. IHG was one of the first companies to run Elastic on a hyper-converged infrastructure.
IHG first came to Elastic for log aggregation. Initially, IHG was running thousands of Linux virtual machines but wasn't collecting syslog data from those machines. To solve this problem, IHG turned to the Elastic Stack, using Filebeat and Winlogbeat to ship logs from application virtual machines and Windows into Logstash. The data was then fed into a messaging queue before being indexed in Elasticsearch. This made it easy to pull up logs in a single window in Kibana rather than dozens of terminal windows. IHG’s adoption of the Elastic Stack signaled a paradigm shift in how the company thought about data — now, there was an expectation that data could be collected in real time and translated into meaningful, actionable dashboards.
IHG’s Elastic platform became popular among the company’s application developer teams, and the amount of data ingested each day increased rapidly. “With that growth came issues,” says Godfrey. In particular, it was clear to Godfrey’s team that IHG’s Elastic platform needed to be expanded, so they began to experiment with their infrastructure to find a solution. “We had some hardware lying around ... converged hardware,” recalls Godfrey. So the team utilized that existing hardware to put together a proof of concept on their corporate cloud. The proof of concept was successful, and they secured funds to make the switch to a hyper-converged infrastructure. They selected Nutanix as their hyper-converged vendor, and worked with Elastic Support to optimize their Elastic deployments on Nutanix nodes.
Now, IHG’s Elastic platform, running on a hyper-converged infrastructure, is managed by a single part-time engineer and can ingest up to 4TB of data per day at both of their deployments, with a maximum index rate of 120,000 messages per second. For IHG, running the Elastic Stack on a hyper-converged infrastructure has been a huge success. “It’s performing. It’s rock solid,” says Godfrey. “It never ever goes down.”
Watch Benjie Godfrey’s full Elastic{ON} Atlanta presentation to learn more about why hyper-converged worked for IHG and the lessons they learned along the way.
