Action-Oriented Metrics for IT Performance Management

by Sateesh Andra

Remember the old adage "If you don't know where you are going, then any road will take you there"? Unfortunately, the "any road" approach -- or worse yet, a "no road" approach -- has for too long been the governing principle of many an IT organization.

Today the CIO cannot survive with an "I really don't know how IT is performing and I am afraid to find out" mentality. More than ever, the business is relying on IT for its success, both operationally and strategically. The enterprise depends upon effective IT management to meet critical business objectives. The business demands nothing less.

The CIO is challenged to maximize the business value of the entire portfolio of IT services by judiciously selecting and prioritizing all IT operational expenditures and investments. Specifically, IT performance management is essential for:

• Connecting IT to the bottom line. This is the ability to demonstrate and communicate that IT services bring real value to the company's bottom line. The performance must be visible and its effects measurable.
• Business/IT alignment. The CIO must ensure that IT spending is directed toward the prioritized needs of the business. That is, IT must be doing the right things, and IT plans must match and support the major objectives outlined in the business plan.
• IT efficiency/effectiveness. When IT spends, it has to get the most for the expenditure. Not only must IT be doing the right things for the business, it must also be doing them as effectively and efficiently as possible.
• General business success. IT needs to support business operational needs and critical strategic challenges.

THE DASHBOARD DEFINED

In its simplest form, a dashboard is a visual representation of data that is normally hidden, such as the engine temperature or the amount of gas remaining in a gas tank. Obviously an IT dashboard is much broader in scope and more complex in its makeup. I define an IT performance dashboard as a visual tool that effectively identifies, collects, measures, and communicates the performance results of key IT services, thereby demonstrating the value and relationship of IT performance to business performance. It is critical that IT performance metrics be tied to both IT and business results.

IDENTIFY THE GOALS

As in any worthwhile endeavor, it is important to understand the rationale behind an IT performance dashboard initiative. Why would a CIO develop an IT dashboard? What results is the IT organization striving to achieve? What are the goals it wishes to accomplish? Correctly answering these questions is the critical first step in the dashboard implementation process. The answers establish the baseline, the foundation for designing the dashboard. There are two important points to remember:

1. Although the usual drivers are common to most businesses (business-IT alignment, IT effectiveness, etc.), the actual components and key metrics of the solution will vary for each IT environment.

2. Dashboards are most useful when they present leading or current indicators, not historical indicators that only measure past performance.

Nonfinancial indicators are more likely than financial measures to show what's coming down the road. They show trends and indicate possible future results based upon the current information.

For example, do you need to establish metrics to demonstrate performance against service-level commitments? If so, operational performance metrics tied to response time and output deliveries will be critical measures. Or perhaps you need to show the linkage of your software development prioritization/delivery process and how that process contributes to the bottom line. Demonstrating development resource allocation to the key business initiatives becomes paramount. Perhaps the annual budget review is a justification nightmare, because no one in the business understands, or appreciates, why an IT investment in infrastructure is necessary. They only see that IT costs are a large percentage of their indirect costs and capital investments, and they therefore challenge every investment based on some intuitive comfort level. Maybe you need to show that IT performs efficiently, gets the most bang for its buck, and that it is supporting the right things for the business. You'll need to demonstrate that IT services contribute true value and benefits to the business, matching IT costs to the business functions that really merit the IT expenditures.

Once you have defined the goals, develop the supporting action plan for each metric. Measurement without action is of no value. It is pointless, and sad to say, it is embarrassing to the CIO if IT meetings take place utilizing dashboards ablaze with "red" indicators that are shrugged off or ignored. If that's the result, the CIO not only missed the boat but should have his or her passport revoked!

UNDERSTAND YOUR COSTS AND ASSETS

CIOs and their management team maximize IT's financial performance by controlling IT spending and evaluating all aspects of IT spending according to bottom-line impact. In order to clearly demonstrate performance, it is imperative to have a firm understanding and grasp of the resources being utilized in the delivery of all IT services. Thus people, facilities, hardware, software, security, administration, and network costs must be identified, understood, justified, and then appropriately tracked, traced, and reported on. This includes both internal and outsourced resources (people and systems) that support operational expenditures and capital investments. This is critical, if for no other reason than to eliminate underutilized, redundant, or unnecessary resources. How can you measure the performance of what you don't know or do not understand? When and if measurement results were ever made known, it would be impossible to make informed judgments and take appropriate action.

DETERMINE THE SCOPE OF THE DASHBOARD

Once you have determined the WHY (objectives) and understand the WHAT (assets and resources), you can focus on the IT services activities you need to measure. The activities can be viewed and categorized by quality, cost, timeliness, efficiency, effectiveness, and usage. IT services are broad and varied. They include data center operational data, security support information, software development information, call center support, vendor/contract management data, and IT strategy and administration support information. Not all of this information will be relevant to the key objectives you have identified. In fact, if you tried to measure it all, the IT organization would be overwhelmed by details and the volume of information. So how do you decide what to measure? Go back to what you identified as important in the objectives analysis phase.

Table 1 shows examples of linking actual measures to stated objectives.

Table 1 -- Linking Objectives to Actual Measures

All the key performance indicators and operational metrics must be chosen for their ability to engender action. Ask the hard question about each individual measure: how will it be used? The normal response is to ensure that performance stays within the acceptable range. If that is the case, then what action is necessary to correct any variances? Defining an action plan to support each measure is critical for success. Measurement without action is of little value.

DETERMINE WHO NEEDS TO KNOW

Once you've identified the activities to measure, determine who needs to know and act upon the information. Who is to get the message? Selecting the right audience to communicate to is just as important as knowing what to communicate. You want the recipient to be proactive and prevent problems before they occur and also be able to react quickly to address the results being conveyed. Different levels within the IT organization will require different summaries and views of the same information. Business partners will also view the same information quite differently than IT professionals will. Therefore, the dashboard has to easily accommodate both groups. An effective performance management system addresses the needs of all players.

The following is a typical list of IT responsibilities. Actual IT organization structures vary, but these functions will be included somewhere in the IT organization:

• Infrastructure (data center operations, call centers)
• Development (process improvement, project/program management)
• Architecture (technical strategy, planning, standards)
• Vendor management (contracts, outsourcing relationships)
• Administration (education, personnel, facilities, security)
• Business unit relations (prioritization, resource allocation, strategy)

IT operational personnel are usually the first line of defense in recognizing and correcting abnormalities reported by the dashboard. Therefore, these individuals require real-time reporting at the detail transaction level of data collection. For example, computer operators/system programmers need to know what the response times of the hardware assets are so they can take appropriate corrective action at the component level and bring them back into compliance with service-level agreements.

Functional managers, on the other hand, deal with more summary-level information. Among the questions they might ask: Has uptime for the network met and supported the 99.9% uptime called for in the service-level agreement for that week? Is the development effort on schedule/budget for the month? Has the amount of project work completed met the forecast for that period?

The CIO is concerned with overall performance and whether it is meeting/exceeding business expectations. The CIO wants to know about ROI, TCO, and resource allocation in support of critical business initiatives. Yes, CIOs also want to know if service levels are being met. But more importantly, they are interested in those performance indicators that link IT value and contribution to the bottom line.

Figure 1 is a simple illustration of the data summary levels and the corresponding organizational components they serve. It illustrates the distinction between transaction data, information, and key performance metrics and their correlation to the different organizational levels that have prime responsibility for taking action.

Figure 1 -- Data summary levels.

DEFINE THE KEY METRICS

It's true what they say: if you can't measure it, you can't manage it. It is essential that you first select the right areas to focus on and then select the right indicators. The selected indicators should tie directly back to and support your defined objectives. The number of metrics you select is also important. Select too many, and you will discourage everyone -- overburdening those who need to know and obscuring the important measures. Select too few, and you will miss identifying performance indicators that are vital to IT and the business.

IT dashboard metrics need to do three things:

1. Answer fundamental questions about IT services. Dashboard metrics should answer basic questions about the overall health of the IT organization. These can be financial or operational in nature; for example, benchmark comparisons (within and outside the industry) and TCO metrics.

2. Alert IT management to issues or problems. Metrics tell the IT management team that something is going wrong, such as excessive application failure rates, budget and/or schedule slippages, and/or poor quality (as evidenced by high call center activity, excessive application errors).

3. Help the CIO make decisions that impact the business. Dashboard metrics provide information that the CIO uses to make strategic decisions:

• Outsource or keep inhouse
• Consolidate or decentralize
• Overall level of customer satisfaction
• Resource allocation

Make sure that the specific performance indicators you select address one or more of the above three areas. The dashboard system needs to collect, summarize, and report to the right people at the right levels within the organization. Select the right measures and then communicate them to the responsible area for action. Here are some specific performance metrics you might consider:

• IT metrics -- capital/operations budget performance, ROI, TCO, schedule/budget performance, resource allocation/usage, error detection, response time, report delivery to schedule, maintenance costs, number of errors, system availability, connect time/utilization, cost efficiency/effectiveness (IT standards and tools)
• Business function metrics -- customer satisfaction indicators, ROI, TCO, service-level performance, application performance, resource allocation to plan

IMPLEMENT PROCESSES TO PRODUCE DESIRED RESULTS

Next you'll need to implement an integrated process to define, collect, summarize, and communicate the information throughout the IT organization (vertically and horizontally). The CIO should view the dashboard as the visual display of an IT enterprise management information system, with all the disciplines and flexibility required by such a comprehensive system. The implemented methodology must clearly identify WHAT, WHO, and HOW for communicating dashboard results. Roles and responsibilities for taking corrective action have to be clearly defined, and performance measurement definitions must include corrective actions. The CIO needs to assume ownership and take an active lead in defining the process and selecting the tools that facilitate the creation of the performance management dashboard. After all, it is the CIO's system!

Some good news -- help is available. Integrated software engines are now able to collect information at the lowest levels from disparate IT databases and bring it together in a common repository for editing, summarization, and communication utilizing dashboard visual aids. However, not all of these tools are created equal. When selecting your software system (tools), ensure that it manifests these essential attributes:

• Easy to integrate. It should have an open architecture that affords seamless integration of data from existing monitoring tools, business applications, relational databases and data warehouses, as well as flat files and XML documents.
• Scalable/reliable. The architecture should be built upon a highly scalable and reliable n-tier architecture designed to grow strategically with an IT organization.
• Secure. The application and technology infrastructure must allow users to build security policies as simple or complex as needed.
• Web-based. Users must be able to access the system easily and securely through the Internet, intranet, or extranet.
• Standards-based. The application and supporting technology must be built with strict compliance to software engineering standards.
• Comprehensive. The system must have the capability to support and report on information from all IT functions.

As with all best practices, it is critical that the process be clearly understood and utilized by all involved. A best practice not followed, or just given lip service, is useless. Worse still, it can create a false sense of security. When trouble strikes, management may take no action or inappropriate action because of an erroneous sense that things are under control and being "managed" at all levels within the organization.

For the IT dashboard to succeed, IT and end-user management and operational personnel first must be educated regarding the process and then trained in the effective use of the dashboard tool. They need to understand the benefits of the dashboard and how to read and react to what the measures are telling them. Over time the process must become institutionalized with the IT organization; that is, it must become the normal way IT operates and responds to the situations conveyed by the dashboard. When it does become necessary to change, you change the process to reflect the new realities.

As with any performance measurement tool, the indicators are only as good as the raw information collected and the relevance of the measurement standards that have been established. A dashboard is an aid -- it is not a substitute for diligent, effective management. You still must keep your eyes on the road.

SUMMARY

Today dashboards have emerged as a key management tool to help the CIO manage the IT enterprise (a business within a business) and demonstrate bottom-line value to the business. Careful planning needs to go into the implementation of a dashboard system. The specific steps that I have highlighted are:

• Define what you want to accomplish: desired goals
• Understand what you do: delivered IT products/services
• Understand your costs: IT assets and resources
• Decide on the key areas that need to be measured: dashboard scope
• Identify who needs to know: horizontal and vertical organizational communication (IT and user)
• Define the performance criteria to be measured: key metrics
• Select tools and implement and institutionalize the process: methodology and tools to facilitate process
• Manage results: proactive management to improve performance

An IT performance dashboard should do two things: (1) tell the CIO and the IT management team how they are performing; and (2) based on that information, influence the responsible party or parties to take appropriate action, corrective or otherwise. Information that is not acted upon represents a wasted opportunity, and such inaction obviously can have dangerous consequences for the organization. The dashboard is there to help the organization keep score and adjust so that informed decisions are made to produce desired results.

One last point: implementing an effective dashboard-based performance management system requires the complete commitment of the CIO and the IT management team. As with all successful process change, executive leadership is essential. The CIO must step forward and demonstrate that performance management within IT is not a "nice to have"; it is essential in today's IT organization.

ABOUT THE AUTHOR

Sateesh Andra is a seasoned entrepreneur and executive who brings vision, strategy, and leadership to MetriKus. In 2005, Mr. Andra founded MetriKus, a leading provider of business intelligence software that allows a single unified view into the IT organization, highlighting key metrics that enable the CIO office to gain end-to-end real-time visibility, develop insights, and drive the overall performance of IT.

As cofounder and CEO of Euclid, a leading provider of IT management software, Mr. Andra raised US $35 million in venture capital funding for the company and was instrumental in taking Euclid from its infancy through a growth phase to 100+ employees. Earlier, Mr. Andra held management positions in business development and marketing in two Silicon Valley startups: RealChip (Comstellar Communications), a VoIP chip company, and Tsqware Inc. (Conexant), a network processor company.

Other than taking a leading role in startups, Mr. Andra gained operational experience in public companies such as LSI Logic and Wipro. He spearheaded development and product management of the Fast/Gigabit Ethernet Chips at LSI Logic that were designed-in at major networking and telecom OEMs, including Cisco, Nortel, and Lucent. Mr. Andra can be reached at sateesh@metrikus.com.