From ELK Stack to Elastic Cloud Enterprise: Scaling Up Capabilities at John Deere
When it comes to farming, most people’s conceptions are outdated. “[T]hey think of some guy in bib overalls driving a tractor,” said Tim Arp, Senior Operations Engineer in the Intelligent Solutions Group (ISG) at John Deere. “But really, it’s all about the data now.” And that data — whether it’s GPS information powering self-driving equipment, metrics about machine performance, or even data about how crops are being planted — represents an immense amount of information pouring into John Deere’s systems.
While many people associate the company with its ubiquitous “green machines,” John Deere’s data-based applications are just as crucial to agricultural professionals. Today, John Deere supports farming operations with machine and agronomic telematics applications that help farmers maximize yield and manage costs.
At Elastic{ON} Tour Chicago, Arp shared how the company began this work with JDLink, their first mobile application. Originally built on 25 servers, JDLink provided farmers with an easy way of collecting data — including geographic location and machine health metrics — on farm equipment such as combines and harvesters. But as the application grew, requiring the deployment of more and more servers, the ISG began to encounter difficulty scaling, ultimately resulting in outages. It quickly became clear that to ensure that the application worked for the farmers who relied on it, the ISG needed a way to aggregate logs and consolidate data so they could visualize the system holistically.
In 2013, the ISG found a solution: the Elastic Stack. Arp and a colleague began experimenting with Elasticsearch, putting together a working demo using version 0.98. Before long, they were able to build their first production cluster on version 1.4, using 4 nodes.
“We started pulling in all kinds of logs: web server logs, application logs, system logs, load balancer logs. What we soon realized was all the analytics that we could get with this. No longer did we look at servers individually, but we looked at the system.” Tim Arp, Senior Operations Engineer, Intelligent Solutions Group (ISG), John Deere
Arp and team also used Kibana to create useful dashboards for monitoring errors and Apache web logs, visualizing response codes and hits by server.
Over the years, John Deere’s application platform has expanded to help farmers in a range of crucial areas. No longer focused solely on data management, the company offers applications to support remote management, guidance, variable rate application, and field and water management. As its offerings have grown, so has the organization’s use of Elasticsearch.
Their current environment, Logcentral@Deere, scaled up from 4 nodes to 25 to provide a logging framework for all of the ISG application development teams. Running Elastic Stack 6.3, the system handles 18 billion documents across 11 terabytes of storage. Roughly 14 different applications log to it at any one time, amounting to 20,000 events being streamed in every second. John Deere has also expanded Logcentral into the cloud, with 8 Elasticsearch clusters. The largest cluster, which has more than 100 applications logging to it, contains 51 nodes.
Today, John Deere relies on Elastic Cloud Enterprise (ECE) for features such as support, integrated authentication and authorization, monitoring and alerts through the Elastic Stack’s alerting features (Watcher), cross cluster search, seamless upgrades, and machine learning. In the future, John Deere looks forward to making use of new capabilities such as rollups and data aggregations to maximize results for the farmers who count on them.
Watch Tim Arp’s full session at Elastic{ON} Tour to learn more about how John Deere is using Elasticsearch to put critical data in the hands of agricultural professionals across the world.
