Kibana platform migration: Lessons in large scale cross-team collaboration

By
Jin Mu

When Kibana 4.0 was created back in 2015, it only had three apps: Dashboard, Visualize, and Discover. Fast forward five years, Kibana now consists of 100+ plugins, millions of lines of code, thousands of dependencies, and dozens of frameworks. The architecture of Kibana that worked well with three apps had become a bottleneck that was hindering Kibana’s stability, scalability, performance, and development velocity.  

In early 2019, the Kibana platform team kicked off the long-awaited Kibana platform migration project. Our goal was to re-architect Kibana’s plugin system, build a Kibana platform to increase the velocity and scalability, and collaborate with 15+ engineering teams to migrate all existing first-party plugins to the platform.

As of June 2020, and after 15 months of hard work, we are proud to announce that the project has reached a successful conclusion. The Kibana platform is in place, millions of lines of code have been restructured, and 100+ plugins have been migrated to the Kibana platform. 🎉🎉🎉

Cross-team projects are complex by nature, and the failure rate is often quite high. We’d like to share what the Kibana platform team learned along the way and hope it will be helpful for engineering managers and project managers who take on similar endeavors. 

Build a collaborative culture 

One of the major challenges when running a cross-team project is the conflicting priorities. Individual teams have their own targets to meet and managers might be reluctant to release key staff members for external projects. Since these conflicts can put the cross-team project at risk, it is key to get buy-in and build a collaborative culture by adopting the following strategies:

  • Explain how the project benefits each team.
  • Get buy in and support from senior leadership. Leaders can help bridge the silos and develop a culture of trust and collaboration by aligning the goals and incentives of the organization as a whole.
  • Show empathy to teams who struggle with restricted resources by expanding deadlines, reducing scopes, or lending a hand.

Set and communicate project goal

Team members are more motivated when they have a good understanding of the project goals and benefits. 

Before starting the migration project, the Kibana platform team spent a good amount of effort on the goal statement and communicating the goal through meetings, Slack, and email. We explained why Kibana needed a platform and how a platform approach would solve the issues Kibana had acquired over the past five years. We also discussed the impact on the developers’ day-to-day experience, how the project goal aligned with the company’s goals, and the value and benefits of a Kibana platform. 

Create a project charter 

Before launching a cross-team project, set up a project charter to outline the mission, responsibilities, processes, and timeline. This will serve as the true North Star for all teams involved. A good project charter covers the following items:

  • Objectives – Establish the project’s main goal, the objectives of each milestone, and problems to solve.
  • Roles and responsibilities – Specify each team member’s role and responsibilities.
  • Procedures – Define the procedures and processes that teams need to follow.

Set up a project timeline and communicate it well

Always avoid an open-ended schedule since it runs the risk of being easy to deprioritize vs. shorter term projects and tasks. A deadline keeps projects on track and creates a sense of urgency. 

After analyzing the project scope, the Kibana platform team set up a clear project timeline and soft deadlines for each milestone. We made sure all teams were aware of the timeline, encouraged them to plan their roadmaps and resources according to the timeline, and frequently reminded teams of the approaching milestone timeline.

Define clear ownership and responsibility

Clear ownership is key for the success of cross-team projects because it facilitates communication. Ambiguous ownership often causes communication breakdowns which is often the biggest reason for cross-team project failure.  Here is how the ownership and roles were defined for the Kibana platform migration project:

  • The Kibana platform team’s responsibility was to build the platform, develop core services, and to provide platform migration guidance and support to participating teams.
  • All Kibana platform team members functioned as platform migration consultants. They were assigned to support three-to-four teams and held accountable for these teams’ success.
  • The tasks of building the platform were divided into multiple tech domains. Kibana platform team members took ownership of these domains based on their technology experiences and strengths.
  • We had one contact person for each team engaged in the project. This person was responsible for leading the team to migrate its plugins to the Kibana platform while adhering to requirements and conventions. Additionally this contact person was responsible for updating the project status, and communicating to Kibana platform consultants when issues arose.  

Standardize the process

Standardized processes can greatly reduce confusion and resource waste. Processes also help teams know who is doing what, when, and how. When it comes to cross-team projects, standardized processes are even more critical due to the involvement of teams from different organizations who may bring their own procedures. Having all teams agree and follow a standardized process helps to streamline the workflow and improve efficiency.

Standardized processes might define the following areas:

  • What meetings would be required and at what cadence
  • Who to reach out to when questions arise
  • How to make decisions
  • How to track project status

Share knowledge

Knowledge should be easily accessible, searchable, and shared on a timely basis to ensure teams have the information they need to make progress. 

Here are some of the actions the Kibana platform team took to ensure effective knowledge sharing:

  • Scheduled meetings for teams to share knowledge. 
  • Hosted training sessions to provide guidance for testing the Kibana platform. 
  • Provided a thorough Kibana platform migration guide and made sure the guide was kept up to date. 
  • Encouraged other teams to contribute their knowledge to shared docs. 
  • Made sure each shared doc had a team responsible for keeping it up to date.

Have a kick-off meeting

Host a kickoff meeting to give all teams and stakeholders an overview of the project goal, schedule, scope, and expectations of all participating teams. Be sure to:

  • Introduce the project team, including which team is leading the project, which teams are participating, and who the main contact person is for each team.
  • Communicate the purpose and scope of the project, the expected major deliverables from each team, the estimated effort, and the timeline.
  • Explain how teams will communicate and work together, how the progress will be tracked, what project management tools will be used, etc.

When people leave the meeting, they should have a common understanding of how the project will be managed, what procedures to follow, which team they can reach out when questions arise, where to find the required knowledge, and what they need to do in both the near and long term. 

Track project progress 

When a cross-team project takes years to complete, you must closely monitor the project status to make sure all teams make steady progress and meet milestones. You must take immediate action when a team needs help.

The Kibana platform migration project was broken into four phases. We asked teams to provide a weekly status update regarding how much progress they made for each phase. We then communicated the status to a broader audience through a weekly email.  

Recognize people and celebrate milestones

People often only celebrate when a project launches and completes. If a project has a long execution period, it is important to recognize and celebrate in-between milestones and major team achievements to keep the team excited and to motivate them for the next milestone. Celebrations and recognitions don’t have to be formal. You can give  people a day off to recharge, or simply send a congratulation message.

Acknowledgement

The Kibana platform migration project is probably one of the most substantial engineering efforts we've ever undertaken at Elastic. Without the teams’ extraordinary collaboration and support, this project would have never been such a huge success. I would like to thank all the people who contributed to this project.

I want to express my sincere gratitude to the fantastic four people in the Kibana platform team: Josh Dover (area lead), Mikhail Shustov, Rudolf Meijering, and Pierre Gayvallet. These four amazing developers put their full heart into this project, and delivered it with high professionalism, strong determination and collaborative spirit. Well done Kibana platform team!