A Fast, Satisfying Search Experience with AmericanAncestors.org
This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.
AmericanAncestors.org, the website of the nonprofit New England Historical Genealogical Society, is one of the leading online family history research sites in the United States. They maintain one terabyte of data in Elastic’s Elasticsearch Service that their more than 250,000 members can search in real time, including over 1.4 billion names and an electronic collection of over 40 million pages of centuries-old, handwritten documents.
The average AmericanAncestors.org member is sixty-five and retired. They have names and locations to research and pages of elegant, faded quill script to decipher. They don’t want to fiddle with a slow, rigid search feature. They want intelligent search that delivers results fast.
Before switching to Elasticsearch, AmericanAncestors.org had used SQL for their search capability, but that solution didn’t have the flexibility their users needed. When Don LeClair, Associate Director of Database Search and Systems, started at AmericanAncestors.org in 2016, they had been running Elasticsearch 1.3 on five physical servers set up by external contractors. A year later, with those contractors long gone, their clusters were no longer optimized, search was slow, and the environment was unstable. LeClair was learning Elastic on the job and, unfortunately, becoming proficient at full cluster restarts.
LeClair and AmericanAncestors.org needed to increase availability and boost performance, but they also wanted to add expertise and reduce system administration time and costs, so they purchased a Platinum subscription to the Elasticsearch Service. Additionally, with their new subscription, they were able to implement Elasticsearch security definitions so queries would come through with appropriate authentication. Another major advantage they found in choosing the Elasticsearch Service was they could create a separate cluster to monitor production and visualize the data in Kibana.
“For the first time we had visibility and facts about how the system was performing rather than opinions.” Don LeClair, Associate Director of Database Search and Systems | AmericanAncestors.org
Once the clusters were optimized, LeClair was able to turn his attention to new challenges. The genealogy research industry was trending toward search that offered suggestions. At the time, AmericanAncestors.org’s approach to search was that results must fit search criteria, but that led to many frustrating member searches that delivered no results.
So they changed to a best fit results approach with prioritized rankings. This change improved the user experience, but also increased the server workload. And that workload increased even more dramatically after Spectre and Meltdown vulnerability fixes were applied to their systems, leading to a performance decline and even an outage. With a new optimization challenge to handle, AmericanAncestors.org decided to work closely with Elastic’s support team to drive improvements.
Through this work, LeClair discovered that Elastic support team’s ethos went deeper than simply working to help mitigate break/fix issues. The Elastic support team helped identify problem queries, performed code reviews on breaking changes between versions, and altered their sharding strategy to increase speed. Support also identified that over 35% of records in their index were logically deleted documents. With this revelation, AmericanAncestry.org changed tools and processes to avoid excessive re-indexing. They also implemented and maintain a full-data test cluster to simplify upgrade testing and enable them to stay current. The test cluster helps during development so AmericanAncestors.org can roll into production with minimal outage and inconvenience.
In the end, with the help of the Elastic team, they built a highly scalable service with minimal overhead. They reduced disk usage by 60% and CPU by 50%, making search 2-3 times faster. This speed enabled AmericanAncestors.org to offer a major promotion with free signup and search for three days. The system handled a 400% traffic increase with excellent response time. “It was … perfect,” said LeClair. “In fact, it was so smooth on the second day I stopped watching it on the monitor.” Their current solution was dramatically simpler than their previous physical servers, and, even with premium support and a full-data test cluster, it cost 25% less than their old solution.
Learn about how AmericanAncestors.org is leveraging Elasticsearch Service to drastically improve their service and the lessons they learned along the way implementing by watching Scalability & Support Using the Elasticsearch Service: Elastic @ NEHGS from Elastic{ON} Tour Boston.
