Opening Statement

by Kenneth Rau, Guest Editor

There has been a love-hate relationship between IT and performance measurement ever since IT emerged from the freight elevator, stumbled into the express elevator, emerged on the executive floor, and became a reluctant member of the business's management team. Accountability became the word of the day, along with budgets, controls, standards, and, eventually, key performance indicators (KPIs) and performance reporting. The "love" came from the natural affinity IT has for data collection and manipulation; the "hate" from seeing commitments quantified and performance results made visible.

Oh, there were exceptions. Some IT organizations quickly embraced the opportunity to quantify every aspect of their operations and report at periodic intervals on progress and procrastinations. But in most cases of IT performance management programs, it became an affair gone bad - initial infatuation, courtship, familiarity, and, eventually, contempt. The excuse used most often for ending the relationship was the effort involved in continually collecting, massaging, and reporting on data that was seldom reviewed and less often used.

In the last few years, there has been a renewed and more pervasive interest in performance management programs in IT. I believe this is the result of two complementary factors. First, in a down economy, ambitious projects tend to give way to cost control and the need to demonstrate value for the expenditure of precious corporate resources. Performance measurement/
reporting as a control program in IT has always flourished in the troughs of economic cycles. Second, we have seen the introduction of business intelligence (BI) software that has removed the drudgery of manual, intensive generation of performance reports. This, in my opinion, is what has accelerated and sustained interest in IT performance programs this time. Both generic tools and tools designed specifically for reporting of the IT function have emerged that enable automatic capture of IT performance data and its presentation in summary form (dashboards) with drill-down capability for the exploration of anomalies.

Let me describe what I mean by an IT performance management program (with the emphasis on "program"). All IT organizations, regardless of size or maturity, measure and report on lots of things - downtime, response time, project status, budget variance, service-level agreements. As the chief provider of performance data for other parts of the organization, it is natural for IT to capture and report on aspects of its own performance. What is different about a performance management program is the concerted, intentional effort to plan and execute a project, the objectives of which are to:

• Understand what is important to measure (hint: those measures that tie together and report on business and IT objectives)
• Set performance goals
• Report on trends and exceptions
• Establish benchmarks to complement reported values
• Routinely and systematically capture, validate, and expand on these metrics
• Establish and participate in performance review meetings at which decisions on required corrective actions are made and subsequently implemented

Sensitivity to organizational change issues regarding the implementation and publication of performance-related data, routine program reviews, and the expansion of the program throughout the organization complete the compendium of things to consider when implementing a comprehensive performance management program.

At the appropriate time in the program's evolution, the addition of an automated tool to capture and report on key metrics is critical to sustaining and legitimizing the program. Prior to the turn of the most recent century, such tools were either crude (read "spreadsheets") or expensive, relying on database design, construction, and population and data warehouse software. The unreliability and drudgery of the former solution led to lost credibility and interest; the cost involved in the latter put it out of the reach of all but the largest IT organizations.

With the advent of BI tools, automation of the IT performance management program became feasible and, dare I say, almost fun for IT. Early adopters piggybacked on the BI tools that their companies had acquired for financial and marketing purposes from vendors such as Cognos, Hyperion, and Informatics. More recently, BI tools designed specifically for IT performance reporting and the generation of CIO dashboards (from vendors such as Pilot Software and MetriKus, whose CEOs are two of our contributing authors this month) have become the tools of choice for automating the collection of data and reporting on IT performance.

However, blind reliance on automation and standard measures as the shortcut solution to IT performance management implementation is folly. As authors Jim Love and Alex Resnick state in their article, "It's not about the tools. Beware the pretty gadgets and cool tools. Keep focused on the measures themselves." The authors also make a strong case for including so-called intangibles among your chosen measures, observing that "qualitative measures are often more indicative of success than the supposedly more 'concrete,' quantifiable information." It's worth taking the time up front to get the right metrics, they say, because carefully chosen measures "can pull an organization into alignment, surface problems, and be a catalyst that propels you to outstanding levels of breakthrough performance."

Echoing Love and Resnick, Jonathan Becher, president and CEO of Pilot Software, recommends carefully considering which metrics are truly important. Becher advocates "mitigating metrics madness" by reducing the profusion of metrics that inundate most businesses today. He cites studies showing that, rather than suffering from a lack of metrics, "organizations are tracking as many as nine times the effectual number of measures." Identifying those metrics that are KPIs is the solution, and the article discusses three tell-tale signs for spotting which of your metrics are key and which are superfluous. Becher argues that most of the metrics we collect, report on, and seldom use are rationalized with such arguments as, "We've always collected this metric" or "We collect it because we can." He points out that "this abundance of metrics can actually be a detriment to organizational success" because it distracts us from identifying and collecting the measures that could provide us with insights into emerging opportunities or threats.

In our next article, Sateesh Andra, CEO of MetriKus, observes that a dashboard is "a visual representation of data that is normally hidden." He expands on this admittedly simplistic definition by further emphasizing that the IT dashboard must be "a visual tool that effectively identifies, collects, measures, and communicates the performance results of key IT services, thereby demonstrating the value and relationship of IT performance to business performance." Andra expands upon the common theme that great care must be exercised in selecting metrics for the dashboard, insisting that chosen metrics must: (1) answer fundamental questions about IT services, (2) alert IT management to issues or problems, and (3) help the CIO make decisions that impact the business. He outlines steps you can take to identify these contributing metrics when implementing a CIO dashboard in your own organization.

Next, Pamela Hollington argues that while metrics such as "percentage uptime" and "number of help desk calls answered" are fine as far as they go, they fail to show how IT contributes value to the business. "'Keeping the environment running' isn't enough for most organizations," she notes. "New business initiatives require new technology solutions, and thus an additional set of measures is required to reflect the CIO's contribution to these projects." Hollington recommends that the CIO work closely with his or her colleagues on the executive team to develop these business value measures, which should be tailored to each project. For a new order and inventory management system, for example, the most salient metric might be "cost per order"; in the case of a point-of-sale solution for a new retail store, "on-time delivery" will be the critical factor. While it is important to report on metrics that demonstrate value, Hollington cautions that the CIO dashboard should not include metrics over which IT and the CIO have little or no control (e.g., new store revenue). She concludes, "There is a tricky balance, therefore, between identifying business value measures and making sure that they are representative of the CIO's contribution to the project and the organization's success."

Paul Williams, past international president of ISACA and its affiliated IT Governance Institute (ITGI), gives us an overview of the latest ITGI initiative, Val IT. Based on COBIT, ITGI's long-standing international framework for good IT governance and control, Val IT is a governance framework supported by a globally developed set of guiding principles and key processes designed to measure whether IT investment decisions are appropriate and deliver value to the organization. Williams contends that following Val IT can help organizations obtain increased visibility into the IT investment decisions they are making, which in turn will lead to fewer mistakes, higher delivered value, and greater confidence in IT's ability to deliver. Val IT's guiding principles require organizations to define specific value delivery practices that monitor key metrics, report and respond quickly to any changes or deviation, and are continually monitored, evaluated, and improved.

In the first five articles of this issue, our authors offered theory, guidance, and advice. In the last article, Lee Imrey discusses their practical application in a case study of the selection and implementation of a CIO dashboard in a large government agency. As seems to be the case in all practical applications of theory, the focus of the case study is a bit of an anomaly. In the general conception of performance measurement and the CIO dashboard, a CIO seeks to understand the performance of all areas of IT. Here a client faces centralized responsibility for systems security in an organizational environment in which systems operation and maintenance are decentralized. Application of performance measurement theory enabled the design and implementation of a system for monitoring system security centrally. The agency's selection and use of a vendor to help tailor and install software to monitor this unique situation should also be of great interest to readers contemplating implementation of a performance management reporting program in their own organization.

And so, just as I began the introduction to this month's journal by stating that a love-hate relationship exists between IT organizations and performance management, there would appear to be a love-hate collaboration going on among some of our authors. All seem to agree that organizations must be very careful in selecting those metrics that are key to depicting IT's performance and that value delivery is the fundamental criterion. From there we have divergence in emphasis and approach. Love and Resnick are leery of too quickly embracing a tool; Andra sees the dashboard as enabling. Becher would focus efforts on extracting the KPIs from the pretenders; Williams would have us focus strictly on tracking value. Hollington cautions against adopting business value measures that IT does not control; Imrey's case study offers a dashboard focused exclusively on organizational security considerations.

I believe that each article in this issue makes important contributions to the body of knowledge surrounding performance management and the CIO dashboard and offers important insights for organizations contemplating their own programs. I did notice, however, that none of the authors explicitly discussed my three key success criteria: (1) that organizations interested in pursuing performance measures and a CIO dashboard should formalize the development effort into a project, (2) that establishing an ongoing performance management review meeting is required for program continuity, and (3) that organization change management considerations are key to successful implementation. In my experience, no performance reporting program has ever succeeded for more than a few months without these three success factors, so I would recommend that you consider them in your own performance management efforts.

About the Author