Building One KPI to Rule Them All

Key performance indicators are critical tools for evaluating how effectively a company is executing strategy and for making optimal decisions. When KPIs are well chosen and well defined, they help everyone focus on the particular business results that matter the most. They can gauge the combined impact of a diverse mix of resources — individuals, teams, products, or the business as a whole — and they can reveal the extent to which those resources are adding value. They can also provide a powerful tool for senior management to influence behavior in the field at scale.

But constructing KPIs that get to the heart of the activities that are most significant in moving the company toward its goals, and that meaningfully reflect performance, can be devilishly hard. In many ways, the search for a comprehensive KPI is the corporate equivalent of chasing down the theory of everything.

A spring 2022 MIT Sloan Management Review article chronicled Agoda’s quest to find the right performance metrics for our business. We have continued our efforts to identify a KPI that could give greater consideration to a companywide alignment of strategic goals while enabling us to adapt to rapid changes in the business environment. We wanted to better account for the range of external factors that lay beyond our control.

In this article, we aim to provide insight into how we worked through our own challenges, in order to provide some guidance to organizations with similar ambitions.

Wanted: A Comprehensive KPI for Business Developers

Agoda is a global marketplace where hotels, airlines, attraction providers, and other organizations can sell inventory to customers. The business development team supports those partners, helping them grow their businesses on the marketplace. The team ensures that the relevant products are available at the requested quantity and at the best price in order to generate more bookings and revenue for us and our partners.

We needed a KPI that could measure how well business developers were delivering on their mission and be implemented as a predictive tool to help them make optimal decisions. This involved very complex, disparate sets of variables for a variety of products, such as accommodations, flights, events, attractions, and more. For example, when business developers work with hotel partners to sell their rooms, they must track room type (such as standard, deluxe, or sea view), room features (such as free cancellation, breakfast included, and nonrefundable), the number of rooms available, dates they’re available, and the selling price agreed upon with the hotel partner.

Prices on travel platforms can change many times a day. That price variance depends on many criteria, such as check-in period; booking date; length of stay; whether the booking is for a single person, couple, or family; and hotel occupancy. Our business developers strive to take the best actions to maximize their portfolio’s revenue, which for a typical business developer may consist of 100 hotels, with the goal of delivering the best availability and pricing for those hotels on all future dates. With the conservative assumption that every hotel has five room types, each with five possible room features, and five pricing segments, each hotel could have 125 availability/price permutations to optimize for every future booking date. That means a typical business developer needs to optimize for 12,500 availability/price permutations for every check-in date across their portfolio, or more than a million permutations for the next 90 check-in days. Without a comprehensive KPI guiding them to the best possible action, their jobs would be impossible.

The Starting Line

The road to a good KPI is often long and usually winding. We have found that it’s best to start with KPIs that have the following attributes:

1. Are highly measurable and not too “noisy.” To reflect real performance, a KPI should be easily counted and reported on, with clear values. Ideally, any natural fluctuations in this measurement should be much smaller than the effect of any actions taken to change it. We call this “controlling your own destiny,” and it is crucial to building a meritocracy and optimizing the business.

2. Align with the overall business value of the company. A KPI must align with a business’s measure of success, such as revenue toward its P&L. The problem is, the more closely a metric is related to a P&L metric, the “noisier” the measurement can become. The goal is to find a good balance.

3. Are understandable. The easier it is to explain the logic behind a KPI and how it works, the easier the adoption and buy-in process will be.

4. Use a comparable currency. Ideally, a KPI will be comparable between individuals and teams, enabling a fair and accurate measurement of investment and compensation. To get a clear picture of the effect of performance, it’s important to standardize the measurement values across teams and to account for any differences.

5. Minimize communication costs. The above factors help ensure clarity, minimize time spent on KPI-related predictions and debates, and maximize time in the field. In short, they deliver value.

In an ideal world, a company would establish only one KPI. Using multiple KPIs increases complexity: The effort to understand how they relate to one another, how to track and manage them, and how to prioritize and incentivize them can cloud what’s ultimately important. In the real world, one KPI rarely meets all of a company’s needs, but that didn’t stop us from trying.

Input KPIs and Output KPIs

KPIs that measure inputs and those that track outputs have both pros and cons. Because there is often a trade-off between the simplicity, actionability, and real performance of a KPI and how close it is to real P&L value, it’s important to understand the beneficial aspects and the problematic aspects of both kinds of KPIs.

Output KPIs measure results — that is, number of bookings or amount of revenue generated — which tend to correlate directly to the P&L. But output KPIs can be heavily influenced by marketing or product actions, seasonality, and other external factors that can’t be controlled by the business developer. In July 2023, for example, we saw an increase in travel interest to Singapore following a concert announcement from Taylor Swift. No analyst could have forecast with high accuracy the impact of such an event. If Agoda’s Singapore Supply team was evaluated based on a revenue KPI, it would have been credited with high performance — which would have been due to Taylor Swift, not its own efforts.

Input KPIs, on the other hand, track available resources and inducements. They are a direct call for business developers to take action — for example, to increase the percentage of hotels in their portfolios that are participating in Black Friday promotions. The call to action is straightforward, likely to be beneficial (resulting in more competitive pricing and additional bookings), and it doesn’t require the business developer to do much analysis or investigation before execution. Problematically, however, it also doesn’t tell us whether the actions generate value as well as an output KPI would. Take the extreme example of a case where the cheapest rates in the market already exist on Agoda for a particular hotel that is also already cost-competitive. Adding another promotion like Black Friday discounts would only dilute revenue for both the hotel and Agoda.

Input KPIs offer several other advantages and disadvantages. They are easily measurable and controllable. They are clear and can be quickly executed, with the potential to learn and adjust rapidly. They don’t require extensive analysis for the team to build an action plan, so they can be adjusted quarterly with minimal disruption. They do not, however, directly represent value creation to the company, and they are therefore not suitable for ROI metrics. Given that different market teams are likely to emphasize different input KPIs, they cannot reliably be used as “comparable currency” at scale. And because input KPIs tend to use a one-size-fits-all approach, they don’t encourage business developers to pitch other, potentially better solutions. A Black Friday promotion, for example, works well as an input KPI for hotels that have a high concentration of European and U.S. customers, for whom Black Friday is an impulse-purchasing driver, but it is a poor indicator for hotels with a high concentration of customers in Southeast Asia.

Output KPIs, on the other hand, align with business value and are useful as both ROI metrics and comparable currencies. They enable teams to align objectives with hotel representatives via shared incentives such as number of bookings or total margin to be generated, as well as with other departments in the company, such as marketing and finance. Output KPIs lend a stronger sense of ownership to the team, enabling it to choose an action plan to generate value. A business developer who has output KPIs feels like the general manager of their portfolio rather than a foot soldier in a chess game.

The biggest downside of output KPIs is their tendency to be noisy, which makes it hard to make forecasts and set targets within a small, fluctuating portfolio of hotels. And, of course, any noise makes it hard to judge the performance of the business developer.

Finally, because output KPIs are not a straightforward call to action, they typically require more time to investigate and find the right set of actions. The result is a lot of actions based on gut feeling, which we refer to as “optimizing random.”

Dealing With Externalities

Our search for the perfect KPI for our supply team had to account for a wide range of factors beyond our control. Market growth, new flight routes, and geopolitical and macroeconomic conditions will all impact a business developer’s performance, as will competitor and partner activities. Numerous externalities are hard to predict. One that we found particularly difficult was COVID-19 recovery by market, and its impact on booking patterns. The Taylor Swift concert in Singapore caused an enormous surge in accommodation searches, but it was little help for the accuracy of our booking forecasts. The travel boom following Japan’s 2015 decision to cancel visa requirements for many countries boosted our Japan team’s output KPIs. However, a closer look into input metrics revealed that the team benefited more from luck, thanks to the visa decision, than from brilliant decision-making on input KPIs.

Internal factors also can be beyond the control of the supply team. Think marketing investment, traffic origin mix, and third-party rates. Internal A/B experimentation on product, where 50% of global hotels are assigned to a condition that is different from a control group’s, often skews revenue or pricing results. To illustrate, one of our most successful programs, Beds Network, enables hotels to distribute their competitive rates to a wide range of global affiliates. Our product team wanted to determine the exact value to the business from Beds Network by running a negative experiment that temporarily deactivated the program in half of all global hotels. This affected the portfolios of some business developers more than others, who saw a negative impact on pricing and revenue generated by their hotels. The KPIs of some of the business developers who were strong performers and had enjoyed a high level of participation in the Beds Network before the experiment suffered disproportionally. This is a typical example of a situation where factors outside a business developer’s control can significantly impact their KPIs, making it hard to assess their performance in an impartial way.

Our Journey to a Single Measure of Business Developer Value

Considering that the business developers’ ultimate goal is generating more bookings and revenue, the most straightforward KPI would simply be revenue. But because revenue is so often complicated by many factors outside their control, we next considered constructing a KPI around pricing competitiveness, measuring our rates against those of our competitors. This, too, has complications, such as the sheer number of price points. If we considered different dates, hotels, platforms, and customer types, we would be looking at millions of price points, which would be difficult to average and hard to digest. Also, publicly available rates are often different from member rates, further reducing the accuracy of the measurement. External fluctuations, such as competitor marketing activity, can also affect this metric, creating unwanted noise. Another option we considered would give the business development team a single input KPI around program adoption by hotel partners. Ordinarily, to secure the best availability and price, business developers have two routes: They can directly negotiate price and availability for specific dates with a hotel, or they can onboard the hotel to one (or more) of our 50-plus partner programs. Those programs are diverse and cater to different partner needs, such as price promotions for last-minute bookers, exposure on high-traffic web pages, or new payment methods for customers. In theory, program adoption results in better outcomes, but once again, this KPI was not as straightforward as we hoped it would be. That’s because not all programs drive the same incremental value, nor are they “stackable” in their value — that is, many programs are not mutually exclusive, or they overlap with one another in terms of benefit.  Some programs, like a Lunar New Year campaign, are relevant only to certain markets. This approach also does not account for hotels’ varying value: A 20% participation rate for top hotels may be more beneficial than 80% participation among the so-called long tail — in Agoda’s case, smaller hotels that typically don’t have many rooms and revenue potential.

Our continued search led to the development of what we call Supply Equity Points (SEP), a measure of value that considers external and internal factors and assigns a value to every business developer action.

The goal of SEP is to support two important decisions for the business developer in the field: It prioritizes which hotel partners they should interact with, based on each partner’s needs and the size of its business, and it ranks the actions the business developer could take with each hotel partner based on the final margin they are expected to generate for Agoda and the partner. This materializes through a task system, where every hotel has a list of tasks, and every task has an SEP value that represents an action the business developer can take, such as activating a promotional program, negotiating a price, or finding room availability for a certain date. The task values refresh every day, and because each task value is expressed in a common unit, SEP, all global tasks are comparable, making it easier to benchmark performance of business developers globally.

SEP solves a complex problem: calculating an exact margin value for every action a business developer takes and removing external factors that might affect the results of their actions. Indeed, understanding how much revenue the company will generate from a hotel signing on to a specific program is a difficult analytical problem. It requires a lot of data to solve at the scale of our operations, and we found that it is best tackled by data scientists with the help of machine learning.

When calculating the expected value of a task, there are a few key dimensions to consider:

1. What is the depth of potential value coming from the task? Variables such as the amount of a discount, and the room types and audiences they target, will change the value.
2. How will this task interact with existing conditions? If the hotel already has some products live, we need to understand how much incremental value a new task adds.
3. How long will this product last? The expected value of a task can vary a lot if the prediction of how long it will take to capture the incremental value is incorrect.
4. Based on past experiences, do we think anything will change once this product is activated? Maybe once the product is live, the hotel will get a lot more bookings, its occupancy will increase, and it will therefore charge a slightly higher price. Here, we aim to predict, at a high level, the post-activation changes we might see.

We model millions of scenarios with AI to understand what would happen if a business developer were to act on one task in the system today, with all of the permutations we know exist. It’s more difficult to measure the value of some tasks, such as onboarding a new hotel to our platform, because we may lack historical data about the hotel. For these tasks, we make some assumptions based on historical data on hotels with similar characteristics as the new hotels. The task value may be less accurate at an individual ticket level, but is more accurate at an aggregate level.

As might be expected, the methods we use to calculate SEP have evolved over the past five years. We began with a global, aggregated, product-level pre- and post-calculation, ignoring all of the characteristics of an individual hotel except for its production value. Today, the calculation systematically models each permutation within a hotel. The value system models the impact on customer conversion rates and Agoda’s associated margin, and also cleans up the noise from seasonality, macroeconomics, and marketing actions. In essence, the model distills the real inherent value of the business developer’s action.

Challenges With SEP

The single currency of supply is not perfect. Three challenges that we routinely contend with are balancing accuracy with predictability, adapting to rapid market changes, and degrading price competitiveness.

Predictability remains a challenge. Our biggest current problem is the trade-off between the accuracy of the SEP decision value and its predictability. Internal and external factors can influence the SEP value of every decision, making it hard to predict what SEP values may be in the future and, consequently, how teams will hit their KPIs.

Because variables can change quickly, there are gaps between predicted and actual SEP values. How a decision plays out in the real world and the results that it yields may not be as positive as predicted, frustrating a business developer who might feel disempowered by the process.

SEP misses some actions and is slow to react to new products. We found that some positive activity is not encouraged or covered by SEP. For example, although face-to-face meetings are encouraged to help business developers form relationships with hotel providers, the value of these meetings is not directly or easily measurable using SEP. It is very hard for our machine learning model to measure the incremental value that a meeting between a business developer and a hotel partner provides to the company. We ended up using other means to encourage our business development team to increase face-to-face meetings. Also, because it requires a lot of analytical firepower and historical data, SEP does not always capture new products that don’t yet have a performance history — and there are many such products in a dynamic environment.

SEP may discourage entrepreneurship. Like many KPIs, SEP can encourage the fulfillment of short-term quotas at the expense of long-term returns. For that reason, it’s important to reconcile big-picture KPIs with long-term strategic bets. This balancing act is achieved using two KPIs that can mitigate SEP’s weaknesses. One, an output KPI, is a price-competitiveness score that we call the Supply Health Score (SHS). The SHS measures Agoda’s prices relative to those of competitors for every hotel and every check-in date in the future. Despite downsides, such as noise resulting from the need to average many external price points (and practical limits on gathering competitive pricing data), SHS serves as a compass of value, capturing actions not yet covered by SEP (like our relative price position). And it encourages our business developers to think outside the box.

We also empowered local management teams to establish “innovation KPIs,” typically strategic input KPIs that may not perform well in terms of SEP but support longer-term value. In Japan, for example, the team developed a plan to acquire ryokans — traditional Japanese inns. Such properties lacked demand and fared badly in SEP values, but business development predicted that with adequate supply, ryokans could attract a customer base. A lack of historical data prevented SEP from assigning an accurate value to the onboarding of ryokan properties, so ryokan onboarding was added as an input KPI outside the SEP framework.

What to Know for Your Own KPI Journey

So where did we land? In the beginning, we hoped to create one KPI for our business developers. In reality, after engaging in trial-and-error that probably frustrated quite a few people on our 1,000-person business development team, we landed on three measures they are incentivized to maximize.

1. SEP is easy to measure and can be completely attributed to the business developers’ actions, but it is somewhat limited as an input KPI. SEP also serves as a check on local management: Even if they want to invest in riskier, long-term strategies, they must also deliver short-term measurable value.
2. SHS is a price position output KPI. It may be noisy and less than 100% controlled by the business developers, but it maintains their entrepreneurship, covers for SEP blind spots, and helps them, as well as management, see the big picture.
3. The innovation KPI, which is decided locally, allows local management to come up with a nontrivial strategy for long-term value.

We also learned a few things that may be of value to other organizational leaders looking for the “right” KPI for their team.

First, the greater the influence of external factors on the output goal, the higher the importance of an input KPI. In a business like real estate, for example, a typical agent KPI is the commission an agent generates, because there are relatively few external factors that can affect the transaction. (It mostly comes down to the real estate agent’s ability to persuade a buyer to purchase a property at the requested price.) At Agoda, our ultimate output goal is the booking made by a customer. The booking is a result of a great many influences, and many of them are outside the control of the business development team. Our SEP KPI helped us reduce the noise from those factors and has enabled us to reward each person’s performance in a way that is, we believe, as impartial as possible.

Second, the greater one’s analytical capabilities, the more control one has over KPI fluctuations. Our SEP development effort required enormous analytical and modeling work to build the best business-impact prediction for every action a business developer could take. Machine learning and AI have helped us remove fluctuations that might affect performance evaluation. In our industry, given the large number of external factors, our quest for the right KPI would likely have failed without the benefit of reliable prediction models.

Finally, complexity is costly. We learned that for a business development team to adopt a KPI, they must understand it well and not feel the need to communicate about it frequently. If the KPI is too complex, is seen as a black box, or fluctuates to the extent that teams can’t link their actions to the resulting metrics, time will be spent discussing and defending the KPI itself rather than focusing on business performance. In other words, the more complex a business development KPI is, and possibly the more comprehensive it is, the less likely it is to be used effectively. Our search for a ruling KPI was challenging, but we came away with a newfound certainty that in our business and businesses like ours, AI is rewriting much of the KPI playbook. The technology is enabling far more accurate predictions of and flexible adjustments to the actions that matter most in a dynamic context — which means that this won’t be our last search for the perfect KPI.