Vol. 19, No. 4, April 2006 | Printer Friendly PDF version

Mitigating Metrics Madness: How to Tell KPIs from Mere Metrics

Why are seminars on measuring IT performance so well attended but implementation of performance management programs so rare? Could it be that we are afraid to manage IT like a business? Tune in next month as we look IT performance management full in the face — and live to tell the tale. Our expert authors will show you how to design a dashboard with leading indicators that help you take action. You’ll discover how to identify true KPIs (and eliminate mere metrics). You’ll learn how the wrong dashboard can destroy performance, demotivate your people, and mask serious problems — and what you can do to avoid this fate. Join us for a lively discussion of ways to measure IT performance so you can manage it.

With more ways than ever to capture information, organizations today have a wealth of metrics at their disposal. The rise of dashboards has provided a neat package in which to display metrics; with the ability to create a limitless number of dashboards -- from enterprise level, to business unit level, department level, all the way down to the individual -- the number of metrics being tracked has skyrocketed. Nowhere has this proliferation become more evident than within IT organizations. The owners of vast amounts of data apparently feel obligated to show others that they are doing something with it.

Yet for many IT organizations, this wealth of metrics yields a decided dearth of useful insight. Metrics have multiplied to the point that they've become not only overwhelming, but also meaningless to those who monitor them. In fact, studies have shown that organizations are tracking as many as nine times the effectual number of measures.

While many IT organizations have fallen into this trap -- with rationales such as "We've always collected this metric" or "We collect it because we can" -- this abundance of metrics can actually be a detriment to organizational success for a number of reasons. Building on existing transactional data often produces a plethora of metrics that lack strategic context, reflect only past behavior, or don't measure an organization's intended outcomes. When metrics are created simply because the data is available, they often bear little connection to organizational objectives. For example, it can be tempting to create a metric for the number of network outages per day. However, that metric alone may not convey enough information if the service-level objective is based on the users' average response time or encourages users to migrate from the call center to self-service.

Historically, organizations have generated reports by running query tools against relational databases and other data stores. Each individual user would generate his or her own report independently, without regard to the common elements. For example, two different reports may each contain revenue, but they might have defined it in different ways (operating vs. cash flow, all products vs. nondiscontinued products, etc.). In recent years, organizations have tried to standardize these elements into metrics by agreeing on definitions and use cases. Unfortunately, the number of metrics in use has ballooned, especially with the proliferation of dashboards. Thus the organization goes from being overwhelmed by reports to being overwhelmed by metrics. At this point many organizations have so many metrics that they no longer know which ones are important.

THE CHALLENGE OF METRICS

The lack of context surrounding data-driven metrics is compounded by the fact that most metrics being used are activity metrics and represent information about the past. For example, even though number of trouble tickets closed and cost per change request are two of the most common IT metrics, % R&D investment leading to service improvement and retention of high-potential staff may be more appropriate metrics for an IT department that is interested in its strategic value to the organization. Relying solely on activity and lagging metrics is akin to driving while looking in the rearview mirror; it limits an organization's ability to drive in its chosen direction.

Moreover, many organizations naturally draw from transactional (e.g., financial, CRM, ERP) systems for their data. Unfortunately, this means that the resulting metrics typically reflect an organization's activity or outputs, not the outcomes it is trying to achieve. Returning to the example above, it might be tempting to measure user satisfaction by creating an activity metric such as percentage of trouble tickets resolved in the same day, because this data is already contained in most service management solutions. However, the number of trouble tickets resolved doesn't directly measure user satisfaction. Instead, it may be more appropriate to regularly survey a portion of the users in order to gauge reported satisfaction. This survey metric, while subjective, more directly reflects the intended outcome.

Even when organizations focus on the right metrics, they often struggle to ensure that the metrics are understood -- how they are calculated, where they came from, and what they mean -- correctly and consistently throughout the organization. Take profitability as an example. Just on the face of it, how do you know whether it is gross or net? Fully burdened or raw cost? Operating or cash flow? Once you establish metrics, be sure that everyone in the organization understands what they are, how they are to be used, how they will be calculated, and from what source the data will come. Documentation is essential here, with formal, standard definitions. Be sure you have the capability to include such documentation with your metrics.

KPIs VS. METRICS: WHAT'S THE DIFFERENCE?

With so many metrics, how do you decide which ones are critical? The answer lies in distinguishing which metrics are actually key performance indicators (KPIs). KPIs are performance metrics explicitly linked to a strategic objective that help an organization translate strategy execution into quantifiable terms. Well-designed KPIs provide quick insight into trends and summary information, while supporting drill-down into more detailed metrics. This allows an organization to see where it's doing well and where it requires improvements and/or course adjustments. Think of KPIs as the yardstick by which success and progress are measured; they are the measures most tightly linked to the organization's success or failure in executing strategy.

All KPIs are metrics, but not all metrics are KPIs. An organization will have many metrics, but few KPIs. While metrics can be a measure of just about anything, KPIs are the measures that matter most. KPIs should also be actionable; if they are not actionable, how can they help you achieve your goals? In other words, don't measure something as a KPI that you can't change with specific actions.

So when is a metric a KPI? KPIs are metrics that are:

1. Outcome-oriented -- tied to an objective

2. Target-based -- have at least one defined time-sensitive target value

3. Rated or graded -- have explicit thresholds that grade the difference (or gap) between the actual value and the target

The above three criteria, used in evaluating whether a metric meets KPI status, serve as a "litmus test" to help ensure focus on the measures that truly matter to the success of your organization. Each of the three criteria is described in further detail below.

THE KEYS TO KPIs: A THREE-PART LITMUS TEST

1. Outcome-Oriented

The first -- and perhaps most critical -- criterion a metric must meet to be deemed a KPI is that it must be outcome-oriented. Metrics that track inputs (the amount of financial and nonfinancial resources applied to providing a service or producing a product) or outputs (the quantity of service or products produced) are just metrics. A KPI tracks outcomes that measure progress toward a defined goal so that you can understand impacts.

Another way of looking at this is that a KPI is explicitly tied to an objective. If you can't describe the business goal it's monitoring, it's not a KPI -- it's a metric. While it might seem natural to simply start by looking at your current pool of metrics and asking which of them meet this criterion, such an approach focuses only on those measures already being tracked, which may exclude other measures critical to your objectives. For example, if you don't currently measure employee satisfaction, then it likely will be excluded from consideration, even though it may be a critical factor in the retention of high-potential staff strategic objective.

A more effective approach is to start with your goals. Take existing metrics (and, as much as possible, take organizational politics off the table for a moment) and ask yourselves, "What measures will tell us if we are on track with the objectives in our strategy plan?" In doing so, be sure to consider a balance of measures -- operational as well as financial, leading as well as lagging, and subjective (qualitative) as well as objective (quantitative).

While financial and other backward-looking lagging indicators provide an important view of how the organization has performed up until now, they offer little visibility into how the organization will perform moving forward. Leading indicators, on the other hand, help forecast future performance, providing critical insight into how today's decisions will impact tomorrow's performance and giving you an opportunity to address issues and/or shift course if necessary. If your goal is to maximize customer lifetime value, a common lagging indicator is total $ purchased per customer, while a leading one could be % customers that buy again within six months of initial purchase. If a high percentage of customers buy again after six months, then it's likely they will continue to buy in the future, which indicates a high lifetime value. Alternatively, a call center that wants to reach the goal be a one-stop shop for all customer interactions might normally use the lagging indicator % cases closed in first contact. However, the leading indicator % of self-help transactions on Web site that are abandoned often provides clearer insight into whether customers are getting their questions answered.

While people naturally tend to think of metrics as quantitative in nature -- pulled from transactional systems, for example -- it is important not to overlook qualitative metrics. Qualitative metrics leverage subjective information that is often critical to performance, such as feedback from important constituents (e.g., employees, customers). For example, with the goal of employees feeling valued, a qualitative survey provides detail and depth that may get to the heart of what's behind the quantitative measure of employee turnover.

In starting with your goal of identifying the right KPIs, you may establish KPIs that aren't currently being tracked. What if you can't get access to the underlying data or if it isn't stored in any database? Rather than relying on the time-consuming and error-prone process of manually collecting data from disparate people, a good performance management system will provide you with a flexible workflow and role-based alerts that manage the process of entering, approving, and publishing KPIs. Be sure not only that you are focusing on the right KPIs, but also that you are tracking all of the KPIs critical to your objectives.

2. Target-Based

The second point to remember when evaluating whether a metric is truly a KPI is that KPIs are more than just numbers. In order to provide a meaningful gauge of progress toward organizational objectives, a metric must have context. Milestone-based targets provide this context.

As an example, let's say that a metric intended to measure user satisfaction currently has a value of 61. Is 61 cause for celebration or concern? If the target were 100 -- as it might be for a test in school (where 90-100 is an A, 80-89 is a B, etc.) -- then 61 would be cause for concern. However, if 61 were one of the best scores in the class, then the teacher might resort to grading on a curve, in which case 61 would be a good value. The only way you can effectively evaluate whether the value is good or bad -- and consequently, contributing to or detracting from the achievement of your objectives -- is by having targets associated with your key performance indicators. Targets represent the value you would like a KPI to be at a specific moment in time.

Rather than relying on a single value for a target, you'll also want to develop incremental milestone targets along the way. Effective KPIs have targets associated with a specific time frame. For example, for the KPI reduce the cost of service, the target for one year might be 12%, with incremental milestones of 0% for the first six months and 2% per month for the remaining six months. The milestone targets can be used during the year to determine whether your organization is on track to reach its ultimate goal target.

Determining appropriate targets for your organization can be a bit of a scavenger hunt. Check business plans, budgets, and product plans for ideas. A manufacturing organization might develop a business plan that includes how many units it plans to produce each month or each production shift. A sales organization typically gives each rep a quota of how much that person should sell each quarter. Budgets specify how much you are allowed to spend. All of these are targets. In some cases, you may need to start out with best guesses. Using these as a starting point, you can refine the targets over time once you have additional experience.

Whatever their source, targets should motivate. When establishing target values, keep in mind that values that are either too easy or too hard to achieve are not motivating. Use "stretch" targets, where near attainment (85%) is deemed success.

3. Rated or Graded

In order to actually gauge performance, you'll need to calculate the gap between the actual and the target values of a KPI. In addition to being outcome-oriented and target-based, a true and effective KPI should also rate or grade the size of this gap. Associating a rating system with KPIs provides a quick and easy-to-understand reading of whether a particular KPI status is good or bad, how on or off target it is.

For example, you can use the traditional "letter grade" system, in which A, B, C, D, or F grades are assigned based on the calculated gap. KPIs achieving 90% or more of their target receive an A grade; 80%-89% of target is a B; 70%-79% of target is a C, and so on. The letter grade system implies that an A is a "stretch target" -- a B grade would be deemed success.

For some IT organizations, the idea of assigning letter grades may clash with their culture. In that case, the above system can be adapted to use less rigid score names such as "Exceptional," "Very Good," "Good," "Needs Improvement," and "Unacceptable." Another alternative is the popular traffic-light metaphor, which uses "dark green," "light green," "yellow," "light red," and "dark red." Whatever system you choose, consider the fact that a range of five thresholds is more likely to be accepted than three, due to people's natural tendency to avoid extremes. With five thresholds, even after discarding the extremes, you still have three useful values with which to gauge progress.

THE VALUE OF KPIs

While migrating from metrics to KPIs may seem like a time-consuming exercise, the benefits far outweigh the costs. The simple exercise of focusing attention on a smaller number of key performance indicators and tying them to the organizational outcomes will help better align IT with its users and customers. In addition, there can be substantial cost savings once the organization is no longer spending resources collecting, transforming, approving, and publishing metrics that aren't as critical as others. Finally, by taking this path, IT organizations can better demonstrate their value and more effectively vocalize their true benefit to the overall organization.

Establishing the right KPIs is just as essential a component of successful strategic execution as defining the right strategy for your organization. Trying to execute without KPIs -- or perhaps worse, the wrong ones -- is akin to taking a road trip to a destination that you've never been to without using a map. If you're lucky enough to ultimately make it to your intended destination, the odds are it will have cost you excessive cycles, wrong turns, and detours -- not to mention frustration -- to get there. Key performance indicators are an important part of the roadmap on your journey to better performance.

ABOUT THE AUTHOR

As Pilot Software's CEO and President, Jonathan D. Becher is tasked with providing the overall strategic direction of the company. Leveraging his 15 years of operational expertise, Mr. Becher leads the company's efforts to deliver relevant and innovative performance management solutions to the market. He is a frequent speaker at industry conferences, an active member of the performance management community, and author of multiple papers on a wide range of subjects. Mr. Becher completed his master of science degree in computer science from Duke University and a bachelor of science degree in computer engineering from the University of Virginia. Mr. Becher can be reached at ceo@pilotsoftware.com.

Mitigating Metrics Madness: How to Tell KPIs from Mere Metrics

Did you enjoy this article?

Talk to Cutter today about membership, including one-on-one guidance from and interaction with Cutter's experts and with their peers, access to research, webinars, podcasts, white papers and more.

Request trial membership