Fairfax Media: Powering the Search for a Modern Media Publisher

The media industry is in the midst of a seismic shake-up rooted in a lowered barrier of entry for new players, as well as changing consumer expectations. Keeping audiences engaged requires understanding how content is consumed, and then rapidly innovating to meet changing needs. For Fairfax Media Limited, staying ahead of the curve has meant replatforming the business and moving search and analytics onto the Elastic Stack — a transition that instantly made their data instantly more usable, enabling them to deliver new products faster.

The transition of Fairfax Media’s business to the Elastic Stack was a greenfield project led by its Metro Engineering team in Sydney. “We’re responsible for keeping everything running, while also building for the future,” said Rob Hill, Head of Operations and Infrastructure at Fairfax Media. “For us, this project was a blank slate and an opportunity to step back and think, right, if we had to build a modern media publisher, what would that look like?”

Competition is increasing in the digital publishing space with a growing number of websites and apps competing for readers alongside more traditional media. Fairfax Media needed to innovate at a rapid rate — far faster than their legacy platform would allow. “As a product focused company, success is measured by time to market of new features and our search infrastructure had become a real bottleneck. It was so complicated that our developers actually didn’t want to go near it. We knew then that things needed to change,” said Michael Lorant, a Senior Systems Engineer who manages Fairfax Media’s public facing infrastructure.

More specifically, Fairfax Media needed a way to build search into new applications and surface the right content to the right audiences. They also wanted greater insights from their log data and analytics on how content was consumed. The Operations and Infrastructure team was already familiar with Elastic, after reading about it in the ThoughtWorks Technology Radar. Based on their research of the Elastic Stack, Fairfax decided to make the switch from its previous solution.

As Fairfax Media began using the Elastic Stack to build more critical services, the business wanted the assurance and support that comes with being an Elastic subscription customer. “We are mostly self sufficient and had the technical know-how to implement the Elastic Stack ourselves. However, when operating a site that has millions of unique visitors each day and requires 100% uptime, there’s a real need for support,” said Lorant.

Fairfax Media also wanted ongoing support from Elastic in helping it to architect and maintain its solution and educate developers. “The Elastic support and product teams have taken a real interest in our journey and facilitated improvements along the way. This has included guidance on how to accurately scale our Elastic cluster and also to protect against data loss when building a highly available cluster with frequent events. The support and advice is extremely clear and specific and certainly above standard,” said Lorant.

Elasticsearch now powers Fairfax Media’s content API at the heart of its business. It provides access to more than one million pieces of content which are searchable by audiences and employees. Content can also be accessed by Fairfax’s suite of media applications and websites to populate features like trending news.

Using Elasticsearch and Kibana, Fairfax Media has built a new analytics capability which provides deep insight into how content is consumed. For instance, the business can now see how long its audience spent on a particular page and how engaged they are based on how far they scrolled down. What’s more, analytics are available just 30 seconds after an individual visits a site. “We capture every metric possible to understand as much as we can about our audience and give them a better experience. This includes improving relevancy of search so we can give them the answers they want in the shortest amount of time,” said Lorant.

Finally, the business has built a centralised logging capability using Elasticsearch, Kibana, Logstash, and Metricbeat for collecting and viewing logs and Kubernetes events. The new capability enables developers to monitor application performance and quickly resolve any issues. For instance, by building a dashboard to monitor HTTP 500 errors in their content delivery network logs, Fairfax Media was able to easily identify an attack on their network and take action right away.

Search is no longer a barrier for their business when developing new products. Developers can quickly and easily enable complex searches across Fairfax Media’s content API, enabling them to build and release applications much faster. With enhanced analytics and logging capabilities, the business can also track and optimise performance of these in near real-time.

“In the last two years, there’s been a profound change in the way we think about digital publishing. We need the capability to respond to expectations around traditional digital channels like websites and apps, but also things like voice-activated news, machine learning, and bot-based interaction,” said Damian Cronan, CTO of Fairfax Metro. “What we’ve achieved with Elastic puts us on a solid footing for that.”