Getting on the Same Page: Dashboard Development from Planning to Implementation

by Jim Love and Alex Resnick

Nobody wants a dashboard. They want a lot of things, but a dashboard isn't one of them. Some are looking for increased performance or productivity. Some want the ability to meet deadlines. Others are looking to manage scarce resources more effectively. Some want to avoid failures -- or at least surprise failures. Or perhaps they simply want to make everyone aware of just how much value they really are adding to the company.

Even the term "dashboard" is not a particularly accurate metaphor. When most of us think of dashboards, we think of a car. That type of dashboard would be most undesirable for corporate management. Car dashboards tell us what has already happened. They tell us the speed we have already reached. They tell us how much fuel we have already used. There are some useful warnings, but they generally come on too late for us to do anything. When the oil light flashes, all we can do is pull over and let the others pass us by. Who'd like to add that capability to their arsenal of management tools?

Tragically, as many companies install "dashboards," that's exactly what they will do. They will get the simplistic dashboard of the automobile when what they need is closer to the sophisticated instrumentation found in the navigation system of an airplane. On an airplane's instrument panel, you can see the current status -- speed, fuel consumption, and the like. But you can also get information that can help guide you to where you need to be in the future. Predictive data from the radar system warns you of rough weather and other hazards ahead. Other instruments monitor wind speed and direction to tell you of the external forces holding you back or pushing you forward.

This may seem like semantics, but it is a crucial distinction. No matter how sophisticated the automobile, none of us would ever try to drive solely by watching the dashboard. Yet on a plane, you actually can fly entirely by instrumentation. In fact, there are times where the instruments are superior to human instinct and observation. Our senses are easily fooled, and our reaction times are simply too slow to be of any use at supersonic speeds. But navigation systems allow teams of pilots and air traffic controllers to coordinate large numbers of aircraft simultaneously and bring them all in for safe landings, day in and day out -- even in the most adverse weather. When the consequences are real, the steady teams that run our airline flights are preferable to the "heroes and zeroes" approach of fiction like Top Gun.

So be careful what you ask for, since you might just get it. How you describe what you need is the most important step in the entire process. If dashboards have a single point of failure, it lies in the original planning and design.

From here on in, when we say "dashboard," assume we're really talking about a navigation system -- since we presume that's what most of you are really looking to implement in your organization. That brings us to a new set of questions. What's your destination? What are you expecting to get from implementing this? How will things be different?

Now take a minute to think about it. Then write it down.

ARE YOU READY?

Not if you skipped the last step, you're not. If you were too busy or thought it was a waste of time, do yourself, your employees, and your company a big favor. Don't implement a dashboard. If you can't find two minutes to imagine what you are trying to accomplish, will you really be able to put in the real time and effort to be successful? If you can't, you risk doing more damage than good.

Why? We start from a simple premise: metrics represent a powerful force. The right metrics, implemented in the right way, can pull an organization into alignment, surface problems, and be a catalyst that propels you to outstanding levels of breakthrough performance. But just as in Star Wars, there is also a "dark side" to the force. The wrong metrics or the wrong implementation can destroy performance, demotivate your people, and mask serious problems until it is too late to deal with them.

Few of us are really short of data. In fact, most of us are inundated by it. The challenge we face is actually to reduce the data, to understand and focus on what is really important. As Taiichi Ono, the founder of Toyota's legendary management system, would say, it's about "learning to see." To see, you have to be prepared to look at things in a new way. And to do that, you have to be prepared to invest the time up front.

CHOOSING THE RIGHT METRICS

The number one and two questions regarding dashboards are, "What should we be measuring?" and, closely related to that, "How do we choose the right metrics?" There are currently three methods for answering these questions.

Method One

Measure everything and explore. This is the method many data warehouse proponents recommend. Buy some big honking machines, analyze the crap out of the data, and you will find some correlations between great performance and something else.

This approach assumes that correlation can be equated with causality -- a fancy way of saying that because two things happen at the same time, one thing causes the other. This is a fallacy that continues to fool us even though we've known it isn't true since the Hawthorne studies of the 1920s. The Hawthorne studies, led by Harvard Business School professor George Elton Mayo, discovered a correlation between working environment and increased production. In one example, researchers increased the quality of the physical working environment for a group of six women, and production went up. To many, that is the same as saying the changes caused the increased productivity. But did they? Apparently not, because when the women were returned to their original poor working conditions, productivity went up again. What caused this? Mayo concluded that the productivity increases had to do with the attention that the workers had received, the measurement that occurred, and the culture that had developed among the group. One thing is certain: the environmental changes were not the real cause.

Method Two

A second method is to come up with some measures, try them out, and find out how they work. This school of thought says that it's better to measure something than nothing.

This approach was recommended during a book promotion tour by a well-known speaker who shall remain nameless (and therefore shameless). When we asked a client who attended the seminar what he would do if the metrics he picked were wrong, he smiled and said that the presenter had covered that -- "You simply change the metrics." When we pointed out that it would most likely be his successor who would make the change, his smile evaporated.

In fairness, method two acknowledges that experimentation and learning need to occur. That's valid. But it also assumes that you have a boss (or shareholder) who is patient enough to let you experiment until you get it right. Even if you have -- to take our airplane analogy one step further -- do you really want to be deciding which instruments you are going to use to fly the plane when the thing is already in flight?

Method Three

There is a third method, which is what we recommend -- and use. Since a great dashboard is really a navigational system, the best way to design and test it is with a map. Only in this case, the topography is the business environment. Picture your strategic goals as the destination. Then map your journey backward from your destination. Think of each of the places you pass through on your way as measurable outcomes or things that have to be accomplished in order to get the end results. Think of external factors like wind and weather as assumptions from your business plan. As we start to imagine our trip in this way, what emerges is a comprehensive set of measurable indicators of success.

We've used the map technique as a paper exercise and as an automated simulation. Sometimes we just use it to frame a discussion. Our best results have come when we've engaged a wide cross-functional group of people who go through this together. Not only do we get better solutions, but, not surprisingly, getting buy-in is much easier since they "own" the metrics that they have developed.

Once You Have a Map, Use It!

Not everyone starts out looking for a "dashboard." For example, one of our clients, an IT department, came to us for help in sorting out their project planning for the coming year. They wanted help in managing their growth and dealing with their resource constraints. After some exploration, they added some additional goals. They needed to give the business users options to demonstrate that the IT department was responsive to their needs. In the end, they needed to show that IT wasn't just an expense. They wanted to demonstrate that they delivered real and measurable business value.

When you are trying to juggle 80-100 client projects with multiple clients and still pay attention to a number of major internal initiatives, flexibility is a huge challenge. Even a small change can have massive impacts when there are a large number of elements to consider. While many companies have systems for monitoring work in progress, few have any real group "planning system." In this particular case, everyone in the planning process tried to keep track of work on his or her own spreadsheet. Some had multiple spreadsheets. Confusion reigned.

To cope, we built a very simple simulator using Microsoft Project. We used the model we created to centralize all the information and develop scenarios, which we tested out and used as a basis for discussions. We were able to manage a lot of data and start to see the key metrics that would be excellent predictors of future success.

The planning went very smoothly. Then we came to the second part -- how would they stay on track? We suggested that some kind of dashboard was required. Their response was that they didn't need a dashboard -- they already had one.

They proceeded to show us a number of reports from the current "dashboard," which essentially reported on project status for ongoing projects. While the graphs, widgets, and colored diagrams were impressive to look at, closer examination revealed that all of the data consisted of "lagging indicators." Many of the critical items that we had explored in our planning were not included at all. Beyond the missing leading indicators, how would this dashboard report on the progress toward their goals? Where was the critical item of "value"? Where did key compliance issues (e.g., Sarbanes-Oxley, CMM®, various quality initiatives) fit in, since these weren't really projects?

We also knew that certain key items (e.g., project charters, specifications) were going to be most indicative of success or failure. Yet these were at best binary measures -- they were complete or not. In fact, a project didn't hit the "radar" until the charter was completed, so this critical step might not even be reflected in their existing dashboard. In the simulation, we had painstakingly mapped a number of assumptions and external factors. A key one was how fast they could acquire, train, and deploy additional resources. Was the resource factor included in the dashboard? Not really -- it showed nothing beyond the amount of these resources that had been consumed at a given point.

We thought that we had a clear understanding of all of the elements that had to be monitored to predict success. We had been encouraged by our discussions about all of these elements. Once the planning was done and the monitoring was to begin, however, the client wanted to go back to their old style of reporting. We were certain that the sophisticated look of the dashboard tool was a factor in this, but there was another force at play as well. Once the planning is done, there is a real and natural eagerness in all groups to move to the world of "execution." Sometimes it is as if planning and delivery exist in two parallel universes.

To the credit of the head of that area, once we pointed this out, he caught on immediately, noting the lack of leading indicators in the current dashboard. But it could have turned out differently, and we took a valuable lesson from the experience. In the future, we will not mistake agreement in planning for commitment in execution.

NARROWING DOWN A WIDE VIEW

When you think about it, it shouldn't come as a great surprise that we can so easily divorce planning from execution. Great plans and great metrics come from thinking outside the box -- from widening your field of view. "Getting things done" is often seen as a process of narrowing your focus to weed out the extraneous.

We believe that both processes are essential. The process of questioning that we go through does generate a lot of potential metrics. Before we can use these effectively, though, we have to manage this list and extract the vital few measures. It can be a daunting task. So how can you narrow it down?

The first step is simply to categorize the information. Which measures are about "performance" and which are about "compliance"? Which measures are predictive as opposed to measuring how things are in the current state? Another classification separates what is "quantifiable" from what is "qualitative." We don't do this to judge the measures, but as a way of understanding them better. For example, too many times dashboard designers reject qualitative information, regarding it as somehow inferior to "hard facts." Yet qualitative measures are often more indicative of success than the supposedly more "concrete," quantifiable information. We've all experienced a project that failed even though it had consistently appeared to meet its schedule and budget. Studies have shown that a more "qualitative" measure like strategic alignment is a far better indicator of potential success [1, 2, 3]. The purpose of categorizing is to understand -- not to judge. To be successful, we may need a mix of different measures.

While we might not judge the categories, we do insist on rigorously questioning each metric within a category. We post a sign in our workshops that says "No Platitudes!" -- and we mean it. We encourage questioning and language that is brutally frank. We want people to ask, "So what? Who cares? What would we really be able to do if we knew this?" Only if a proposed metric survives this test does it have a chance at becoming a performance measure.

One question that comes up frequently is, "What is the optimal number of measures?" You want a manageable number. You can't focus on everything, because that's not focusing. If the process of categorizing and questioning is done well, the number of measures tends to take care of itself. The vital few survive, and others are consolidated into composite measures.

To illustrate where we want to get to without specifying the precise number of metrics, we point out that one of the best "dashboards" we've ever encountered appears to have only one measure. It's the one that says, "We've had X accident-free days." Inevitably it is right out on a factory's front lawn, where everyone can see it. It presents as one number, but it's really a composite of a number of behaviors and actions -- a mixture of qualitative and quantitative measures. Yet it exists as a single strategic measure and focuses people on what is truly important. It says, "This is not lip service. We care about this."

The bottom line? Categorize to understand the metrics and focus on what's really important by asking tough questions. Without setting any limits, you will see a set of measures emerge that, with a bit of additional work, can be very relevant and very powerful but still fit onto a single page.

THE "INTANGIBLES" DILEMMA

Once you have decided what is important, you then have to establish a fair and open method of measuring it. Then you have to communicate it widely for it to have any effect. This is a challenge when it comes to qualitative information or so-called intangibles.

Study after study shows that devoting attention to intangibles pays off. In his study of companies that had gone from lackluster performance to the top of their respective industries, Jim Collins included "passion" as one of the key areas for strategic measurement [1]. Last year, a study published in the Harvard Business Review [3] showed once again the link between qualitative measures and results. The authors found that attrition rates for customers were much lower when the customer was "emotionally satisfied"; that is, when they felt good about the transactional encounters they had with bank staff. Where the customer was "rationally satisfied" (all quantitative performance criteria were met -- the traditional "dashboard measures"), the rate of attrition was about the same for satisfied and unsatisfied customers! In one of the early, pivotal studies on portfolio management, a study of over 200 companies determined that those who focused purely on quantitative financial measures had portfolios of projects that were not well aligned with business objectives. In fact, the most quantitative measure (financial) was determined to be the "weakest of all measures." The best single measure from the same study was an "intangible" -- the measurement of alignment with business strategy [2].

So why, given all of this proof, are intangible measures given short shrift? Why do we get the sense that even when they are included, their inclusion is an afterthought -- lip service?

In a recent scorecard that we did, we identified a series of qualitative measures that we knew contributed to superior performance. As we always do when we deal with intangibles, we had good evidence of their efficacy. Naturally, we were shocked when they were dropped. Why did this happen? It turns out there was a fear that if these measures were implemented, they could be used to challenge management decisions -- particularly those relating to promotion and bonuses.

Politicizing measures is all too common. While it's not impossible to do this with quantitative measures, qualitative ones are more vulnerable to political machinations -- manipulation, exclusion, or irrelevant inclusion. The damage this does is extreme. When the metrics are politicized, everyone knows. Trust breaks down, and metrics lose their ability to motivate.

So how do you deal with these challenges? First, you need to get acknowledgment that just because a measure is intangible does not mean that it is unmeasurable. Apply a real rigor to intangibles. Get agreement on how you will measure them before you start measuring. Above all, create the ability for your group to question. Establish a brutal level of honesty early, and the problem of intangibles will be lessened. Performance, not politics, is what should determine your metrics.

THE DATA MYTH

In our experience, companies all too often dismiss performance measures because of concerns about accuracy and completeness of data. We actually had one project that was quietly abandoned after we had begun because it was felt that that we couldn't get "accurate data." Even where data gathering is more successful, there are often concerns that what we are collecting won't have value because there is no historical data to compare it to.

It would be great to have perfect data and even better to have historical data. It might tell us additional things and give us new insights. But you can't give up on metrics because what you have isn't perfect. Nor should we subject ourselves to a greater standard than is necessary to get the results we need.

We frequently have to drive home the point that we are not preparing financial statements. We are looking for information to help us make management decisions. Polls, for example, are accurate to within four percentage points 19 times out of 20. Some have been stunningly inaccurate. Yet they remain a key tool in both marketing and politics. Why? The question is not whether the data is accurate but whether it is accurate enough.

Once again, a successful dashboard results from intense and bold questioning. Would we make any different decision if the accuracy of the data were greater? What is the threshold of materiality where our decision or actions would change? What's more important -- speed or absolute accuracy? Isn't having the exact answer too late to make a key decision worse than making the right decision with imprecise information?

If the data is really important to your decisions, you will find a way to measure it -- or as much of it as you can. The system doesn't need to be elegant; it needs to be useful. And it doesn't even have to be complete. Pareto's Law states that 80% of the problems in a system or process are caused by 20% of the activities within that process. If we are measuring the right things, we can have the same impact with a subset of the data.

We believe that most often questions about data accuracy, data completeness, or historical data are red herrings. When we start arguing about the precision of metrics, we frequently find we have the wrong measures or the wrong presentation, or we are dealing with resistance that has nothing to do with the accuracy of the numbers. Measurement is supposed to make you uncomfortable. Questioning the accuracy of the numbers is how some people vent that discomfort.

Remember the IT department client we mentioned above? When we began replacing their plethora of spreadsheets with a version of the planning module we outlined earlier, we cautioned the consultant who was working on the model to remain calm when (not if) the accuracy of the data was attacked. Very early into a meeting on the new model, one of the participants began vigorously comparing our data to her spreadsheet and pointing out a series of minor discrepancies. The behavior was disruptive, but because we were prepared for this resistance, we stayed calm and didn't get into a defensive position or an argument. We quietly pointed out that the discrepancies were there because we were all working from a number of unrelated spreadsheets.

Prepare for arguments about data. Agree on the margin of error and what you will do about inaccuracies. Do this before you show your data or the results. Even then, beware the smokescreen and anticipate resistance. Metrics hold up a mirror, and the picture we see isn't always flattering. It can get emotional. Remember, if your metrics aren't making someone uncomfortable, they probably aren't the right metrics.

AUTOMATION: A BLESSING AND A CURSE

Automating the collection and calculation of data is often a necessity. After all, measures may increase productivity, but measuring itself is not productive. It can take hours and even days to assemble and analyze information. So you have to consider automation. But it's not about the tools. Beware the pretty gadgets and cool tools. Keep focused on the measures themselves.

Sometimes simple and low-tech are the best way to go. We have seen solutions that cost hundreds of dollars deliver the same value as systems that cost hundreds of thousands. We have one scorecard with 10 values that are simply selected from a list on a monthly basis. Yes, we could have built an enormous system to automatically make that selection, but a knowledgeable administrator can do the same thing quickly and with a high degree of accuracy.

On the other hand, beware the homegrown Excel wizard. We've seen some systems built with a proliferation of spreadsheets, and while they appeared to be "money savers" at first, they ultimately cost more than the value they delivered. Often they were delivered quickly, but many were built on a house of cards -- undocumented macros that delivered results no one could justify. Even if you found the error, the safest presumption was that you'd never be able to fix it without rewriting the whole spreadsheet. That may sound extreme, but it's far more common than you might think.

Here's the best advice we can give you on technology. We actually prefer to build rather than buy, because the tools out there are often far too expensive for what you get or what you need. If push comes to shove, Alex and I will build scorecards using Excel to prototype. If you ignore spreadsheets and off-the-shelf tools, someone else will build it while you aren't there. So use Excel and quick tools to pilot, but move off them very quickly.

When it comes to developing for real, we prefer to use tools such as SQL Server, Analysis Services, and Visual Studio. We do this not because we worship Microsoft, but because these are the tools that most people have lying around, and they are "ready for prime time." If you stick to a standard (and, like it or not, Microsoft represents a standard), it is a lot easier when you want to add something new. We once wanted to incorporate some data from Microsoft Project for a scorecard. Microsoft Project Server is XML compliant -- it allows access to a great deal of information through XML tags. Once we discovered this, we were able to extract and produce some very sophisticated scorecard reporting from real project data in a matter of hours.

At the high end, we are now working on much more complex real-time scorecards using Panorama software, one of a growing list of very reasonably priced and high-performance business intelligence (BI) tools. Panorama uses the same cube that Microsoft uses for Analysis Services, and the software has some things that make life very easy for developers. We can do in hours what takes weeks in other tools. Still, we only recommend it if the client doesn't have an existing BI tool or is unhappy with their existing toolset. Our advice is to stick with the tools you have unless you prove they can't do the job.

The worst dashboards we have seen are the ones that looked the coolest. In contrast, the best dashboard we've ever done is one page, with simple graphs and more text than you might think. But it offered valuable insight and great information and helped deliver results.

Someone, sometime, is going to try to sell you on a feature of a software solution before you've defined what you really need. So ban all demos and even discussions of widgets and gizmos until you are absolutely sure what you want to see. If anyone tries to demo them to you, fend them off with tough questions: "So what?" "Who cares?"

GETTING ON THE SAME PAGE

If you've ever been in the cockpit of a commercial aircraft, you were probably struck by the sheer number of gauges and dials. Yet if you watch a skilled pilot, you see that they focus on specific instruments very closely, while the rest is peripheral. They've learned -- indeed they have been trained -- to tune out the noise.

In the corporate world, we all receive a constant barrage of data. The beauty of a well-done dashboard is that it helps us tune out the noise. It forces us to identify and then to focus on what is really important. We make tough choices on what we show and what we don't. And when it's well done, there is no better way to communicate strategy.

Because of this, as much as we eschew gadgets, we do pay a lot of attention to dashboard presentation and how we bring the information together. We spend a great deal of time thinking about the design elements. We use color to convey messages -- green, yellow, and red. We ensure that there is a flow. We want it to tell a story. Everything is evaluated in that context. We don't care about gizmos, but we do care enormously about clean design and ensuring that the first impression is not information overload (see Figure 1).

Figure 1 -- A sample dashboard showing the typical assembly of components. It has gauges and colored indicators and tables of data that allow you to drill down to specific areas of interest. But while the technology can be helpful, it is not where the value comes from. The real value comes from measuring the right things and presenting the data in a way that can lead to insight -- and action.

We also continually struggle to get it all on one page. One way to do that is to have multiple dashboards. That makes some degree of sense in that what you need to see on the factory floor may not be what is needed in the executive suite. Yet what we want to end up with is a way of linking the various measures and putting the resulting information into one coherent, cross-functional story.

It's a tall order. But it's a beginner's mistake to focus on any individual metric without considering its relationship to the others. Time and again, we see that it's cross-functional measures that enable breakthrough results. A few years back we came up with a composite measure that we call a "performance index." It was our way of meeting two simultaneous objectives: reducing the number of measures and preserving the relationship between different functional areas. We started with the same approach as a stock index. We put together a composite of weighted quantitative and qualitative data. We used it to show trends and projections and to tell a consistent story. Like a stock index, the story is told by the composite, but drill-down techniques allow people to get to deeper levels if they need to.

This approach has solved some additional issues. For example, we have one performance index model that is in its second year. When the client wanted to change a few of the individual measures, we realized that we could do that without losing the ability to compare to previous periods. Even in those cases where there is no history, the movements of a well-constructed index can still provide some valuable insight well beyond what a single metric would do.

MAKING HARD WORK LOOK EASY

In our introduction, we said that developing the right performance metrics was hard work. We meant it. Yet at the end of the process, if it's done well, it should look easy. It should be intuitive. It should get everyone on the same page. It should transform the complexity of vast amounts of data into true insight -- insight that enables decisions that in turn generate breakthrough performance. That performance is what we were looking for in the first place.

After all, nobody really wants a dashboard.

REFERENCES

1. Collins, Jim. Good to Great. HarperCollins, 2001.

2. Cooper, Robert G., Scott J. Edgett, Elko J. Kleinschmidt. Portfolio Management for New Products. 2nd edition. Perseus Press, 2001.

3. Fleming, John H., Curt Coffman, and James K. Harter. "Manage Your Human Sigma." Harvard Business Review, July-August 2005.

ABOUT THE AUTHORS

Jim Love is a founder and Managing Partner of Performance Advantage, a consulting firm that specializes in implementing strategies that increase corporate and departmental performance. Prior to founding Performance Advantage, Mr. Love headed the global community of practice that dealt with governance and value at DMR Group and its successor, Fujitsu Consulting. That group went on to pioneer a number of innovative approaches to corporate and IT governance. Much of their work is described in the best-selling book The Information Paradox. Mr. Love's clients have included large companies such as Bell Canada, Inco, Cisco, Royal Bank, and a host of others. He is a frequent speaker at industry conferences. Mr. Love can be reached at jim.love@performanceadvantage.ca.

Alex Resnick is the President and founder of The Catalytics Group Inc., a management consulting firm specializing in improving strategic execution through enhanced use of technology. Mr. Resnick is a Chartered Accountant with over 30 years of international business and technology consulting experience. He is a published author and respected speaker at international conferences.

Mr. Resnick and his management team built a highly successful technology-based consulting firm that merged with Deloitte in mid-2000. He joined the firm as a partner but found the draw of entrepreneurial leadership too strong to ignore. The Catalytics Group is the outcome of that passion. Mr. Resnick can be reached at alex.resnick@catalytics.net.