CIO Dashboards: Flying by Instrumentation

by Lee Imrey

CIO DASHBOARDS IN THE PUBLIC SECTOR

As legislation requirements impose higher regulatory overhead on government processes, US federal agencies are adopting cutting-edge business management tools to facilitate executive decision making and to increase the value received for each tax dollar. One of the most significant tools is the CIO dashboard, which provides a broader view of world-spanning business processes, while providing the capability to drill down to inspect systems at a highly granular level. Several federal agencies have implemented their own versions of CIO dashboards to monitor and support regulatory compliance efforts. In examining the implementation and ongoing rollout of one such agency's dashboard, I will identify benefits that dashboards may offer, challenges to dashboard implementation, and some of the inherent limits in dashboard functionality.

THE EVOLUTION OF IT AND THE CIO

During the transition of developed countries to an "information economy," the role of the chief information officer (CIO) has evolved significantly. Most readers will remember when a company's data center manager, a position of relative anonymity, was the closest thing to a CIO. The manager was charged with supervising data center operators and ensuring proper staffing and maintenance of a single server to meet the company's operational needs. Today, the CIO is entrusted with all of the corporation's information resources, including data stores, data processing facilities (including e-commerce support services), and e-mail, Web access, and other network services.

CIO responsibilities continue to expand as corporate value is more and more inextricably tied to intangible information assets. More effective data acquisition processes are developed, as well as more sophisticated analytics. Both of these evolving capabilities drive the collection of massive data repositories and allow companies to target markets more efficiently, reduce the cost of supplies, streamline operations, and improve demand forecasting capabilities. In each case, the value of the information-based intangibles increases as a percentage of overall corporate value.

Companies, individuals, and legislative bodies all recognize the value of this data. Some companies try to capitalize on the intangibles in their own production or sales processes, while others package and sell data to other companies. The latent value of the data is only realized when it is consistently available for business processes.

HOW can WE MEASURE THE VALUE OF IT?

The effective application of computer-based information is of critical value to most organizations. However, few can agree on how to quantify the value. How can we measure this value accurately and consistently? How can we show savings attributable to an organization's investment in IT technology, implementation, staffing, and support -- and over what time frame? What return is realized in productivity gains directly attributable to the IT investment? What is the earned value added (EVA), if any?

To provide more objective measurements of business value and to help organizations drive improvements, an evolving set of business performance measurement (BPM) techniques have been developed and implemented. These include total quality management (TQM), Motorola's Six Sigma, EVA, and balanced scorecards. Each technique offers its own strengths. TQM and its successor, Six Sigma, help companies bring greater efficiencies to repeatable processes and reap the benefits of cost savings, while EVA helps companies allocate their resources more effectively. Balanced scorecards provide a framework for analyzing TQM and EVA, together with activity-based costing (ABC), to measure the impact of proposed business changes from multiple perspectives.

KPIs: TYING IT TO BUSINESS PROCESSES

All of these techniques rely upon the selection, measurement, and storage of key performance indicators (KPIs) -- specific measures selected to correlate directly with organizational progress in high-impact areas (e.g., minimum criteria for meeting financial targets, complying with legislative guidelines, fulfilling contractual requirements). Achieving predefined measures on a KPI may determine whether an organization has met the terms of a contractual agreement. Performance on KPIs may measure critical success factors (CSFs) with internal ramifications (e.g., executive compensation) or external consequences (e.g., breach of contract for failure to meet a service-level agreement [SLA]). For any clearly delineated business objective, KPIs are intended to provide a "line of sight" toward successful completion.

However, in today's complex business world, business objectives may be hard to clearly delineate. Businesses may collect a staggering amount of data from a diverse set of systems. KPIs may be broad in scope, relating to the organization as a whole (e.g., stock price), or they may be narrow in scope, relating to specific subcomponents of the organization (e.g., average time to resolution in a call center). There may be too many KPIs to independently manage within an organization, or they may interact in complex and surprising ways.

While complex, KPIs are worth monitoring, in the same way that complex instruments in an airplane cockpit are worth monitoring. Without watching for changes in business-critical measures, a business will crash as surely as an aircraft. But just as in a large aircraft, the volume and complexity of data are intimidating.

CIO dashboards reduce the volume and complexity of data, providing the CIO with a simpler, abstracted set of measures by which to judge success in meeting organizational goals (represented as KPIs and CSFs). Historically, a CIO might pore through the previous six months' worth of reports from help desk management to determine whether help desk staffing is adequate, or he might manually track planned and unplanned downtimes in an Excel spreadsheet to determine whether a telco provider is meeting its contractual SLA. By defining KPIs for specific processes, they can be tracked as a single metric, and a target for that metric can be set (perhaps 60% help desk utilization). This will help in making staffing decisions, allowing the company to manage costs more effectively. Similarly, system uptime during business hours might be measured and specified as an appropriate KPI for tracking the telco's SLA performance. Each indicator can be represented in a fashion appropriate to the function it is measuring, such as pass-fail, scheduling, percent-completion, or other measures.

By having these indicators clearly and concisely presented in the CIO dashboard, senior management will be able to track IT performance against documented business goals. And organizations will be able to track senior management's performance against a quantifiable set of performance criteria.

AGREEING TO KPIs

CIO dashboards offer their greatest value when the organization standardizes on specific KPIs, as determined by the CIO and other stakeholders. This is a critical stage in the development of the dashboard. As informational graphics expert Edward Tufte is quick to point out, it is not just the presentation of data that matters -- it is the presentation of the right data. Sufficient time should be allocated to selecting appropriate measures, keyed to the most significant business processes.

Whether the common criteria are decided through negotiation or by fiat, it is critical that all parties involved accept them, as they will be the basis of the business data the dashboard provides. If stakeholders do not buy into the selected criteria, they may sabotage the measures. They may simply not align their performance goals to the KPIs. Worse, they may try to "game the system," selecting actions specifically to influence the KPIs rather than the process the KPIs are intended to measure.

Once criteria are selected, the CIO dashboard provides a set of visualization and tracking tools for executive management. These views can be personalized to each management role, summarizing specific KPIs for which the role is responsible. It is a real-time performance evaluation against a set of objective measures to which (presumably) the executive and her staff are held accountable. The CIO dashboard offers a high-level overview of IT system implementation efforts and performance measures, allowing the CIO and her peers to see at a glance how well even the most complex set of technologies meets their business objectives.

HOW CAN WE IMPROVE PERFORMANCE?

Dashboards have the ability to produce exception reports, which answer the question, "Why wasn't measured performance higher?" and they provide specific guidance on how to manage KPIs that fall below expectations. Some will also provide the capability to drill down to the data behind the KPIs. While the level of detail on a dashboard is usually limited to summary data, appropriate for a CIO or other high-level executive, in some cases more granular detail may be required. This was the case for the agency discussed below.

A DASHBOARD FOR A FEDERAL AGENCY

To understand a dashboard implementation, we must first examine the organizational context. The agency in question has over 100,000 employees and contractors distributed across hundreds of offices all over the world. Composed of numerous semi-autonomous organizations, it is even more complex than a comparably distributed commercial organization. While private-sector organizations may have as many departments, they typically report through a single hierarchical structure. In contrast, within the agency, each organization has its own CIO, IT systems, and support staff. Some organizations use primarily federal employees, employing contractors only as needed, while others are contractor-based and managed by a federal contracting officer. These thousands of contractors represent dozens of different companies and are employed through a variety of contracting vehicles.

Programs are managed at the component level. Programs must comply with component-level policy and with broader agency-level IT policy, mandated by the Office of the CIO (OCIO). This latter requirement is enforced through the budgetary authority of the OCIO, much as a private-sector CIO might exercise budgetary authority over an IT portfolio.

Requirements for the Agency Dashboard

The development of any dashboard begins with an analysis of customer needs. The agency's needs were complex. It required a means to establish transparency of information security processes throughout hundreds of programs under multiple management hierarchies. For each program, there are hundreds of discrete requirements having to do with policy compliance. In fact, across the agency, there are over a quarter-million auditable policy requirements, each of which must be met or explicitly waived, documented, and validated by the OCIO.

Any of these requirements may be called out by the Office of the Inspector General (IG) for secondary validation. While the creation of system documentation for the OCIO and Office of Management and Budget (OMB) is only required every three years, every organization is required to conduct self-assessments on an annual basis, including every system in every component. Each of these self-assessments may be audited by the IG on an annual basis, although only a subset are audited each year.

The OCIO needed a process to economically collect data on control compliance from each of the components, in spite of the diversity of disconnected systems in use throughout the components. Given the auditable nature of each data point, and the provision that compliance should be demonstrable at a granular level, this represented a staggering amount of data. The agency also needed a means to summarize its status in meeting each of the specified requirements, at an appropriate level of abstraction to allow management oversight by the OCIO but with the capability to drill down to a more granular level when required.

A senior staffer within the OCIO was assigned to manage this process across the agency. Faced with enormous responsibility but with limited resources, he first established a common set of performance metrics for the reporting organizations. The metrics were selected to represent agency, organization, and system compliance with the regulatory mandates and guidance under which each operated.

Looking at Market Offerings

To manage this diverse environment, the senior staffer selected a private-sector company to partner with the agency and customize a solution to meet specific requirements. According to interviews with their president, the selected company had been working on similar projects and saw an opportunity to adapt an existing product to the agency's needs. Their flagship product was a high-level dashboard, customized to track deliverables for another cabinet-level agency's certification and accreditation (C&A) process. It tracked the C&A process for all systems that had been approved for operation.

However, when the dashboard was demonstrated, it was not sufficiently granular to meet the current agency's requirements. The senior staffer had envisioned a tool that would provide an executive overview of the security posture across the agency, including compliance with every one of the major control families specified as requirements for regulatory compliance. The core product dashboard was designed for a higher level of abstraction.

A Complex and Evolving Set of Requirements

The agency required a high level of abstraction, but it also needed the ability to produce more detailed reports. Rather than a single CIO dashboard, the agency and all its suborganizations required a set of "cascading dashboards," which allow summary data to be viewed from the strategic to the tactical level by the agency CIO, the component CIO, or the system-level information system security manager. Each level had to incorporate independent sets of requirements derived from multiple source documents, including:

• NIST Special Publication (SP) 800-26 Security Self-Assessment Guide for Information Technology Systems
• Director of Central Intelligence Directive 6/3
• The agency's IT security guidelines
• The agency's security program operations manual
• Component- and system-specific information security policy

To complicate matters further, the requirements keep evolving. The technologies, threats, and controls change rapidly enough that the regulatory guidance has to change just to keep up. Today the agency IT security guidelines have already been updated, enhancing and expanding upon earlier policy.

Similarly, NIST SP 800-26 is being replaced by NIST SP 800-53 Recommended Security Controls for Federal Information Systems. Currently, the dashboard is being updated to map to the newer regulatory guidance. This will be an ongoing process. NIST SP 800-53 is only intended as interim guidance and will be replaced by Federal Information Processing Standard (FIPS) 200, Minimum Security Controls for Federal Information Systems, which was released in March 2006. According to NIST, FIPS 200 will continue to be enhanced annually, as the threat environment changes. Even standing policies may constitute moving targets.

The senior staffer's understanding of the changing regulatory environment was evident in his description of a highly dynamic dashboard model. He described "a more tactical dashboard," in which security performance could be tied directly to regulatory requirements, with every control traceable to specific documented policy. The agency's CIO dashboard would have to provide "a more realistic representation of [the agency's] security performance" by enabling a deeper view into operations at the component level. In this way, it would combine features from traditional CIO dashboards with those of more operational controls.

Investing Resources in Adding Capabilities

With significant effort, the required functionality has been added to the vendor's product to update it to the agency's current policy, as well as to make it more granular (to provide assurance that policy requirements are met agency-wide). This process has required substantial resource investment from the vendor, the OCIO, and component staff.

The principal efforts have been to align the KPIs with policy requirements. The policies are general in nature and do not always map to objective technical controls. This has required a good deal of subjective judgment, as well as negotiation with component representatives. As discussed earlier, the initial selection of KPIs is a critical process, particularly when they will be applied broadly across a diverse organization. KPIs must be continually defined and refined to meet evolving sets of requirements.

For each requirement, associated agency control, and dashboard KPI, a set of subject matter expert guidance has been developed, providing examples of how to meet control requirements. This has required input from various authorities within the agency and the summarization and delivery of draft guidance to component representatives. As feedback is received, this guidance is updated and integrated into the product to provide more effective guidance to those responsible for the controls the dashboard tracks.

Challenges

Incentives

The responsibility for maintaining the agency's hundreds of secure systems has been centralized in the OCIO. This is similar to large private-sector corporations, where responsibility is ultimately centralized in the officers of the corporation. However, the implementation and operational maintenance of systems were decentralized and delegated to individual components.

This causes an inherent incentive imbalance. While the OCIO is responsible for the security of the systems, there are many factors impacting that security. The incentives may be rebalanced through a weighting of controls, which were only roughly weighted in the initial implementation. Over time, the incentives may be changed through weighting the KPIs in order to constructively influence component behavior.

Stakeholder Acceptance and Support

Achieving and maintaining stakeholder buy-in is another critical issue. "Selling" the program began prior to development and will continue throughout implementation. Until stakeholders derive independent value for their own organization through the dashboard, there is a risk that component-provided data may be suspect. Users of the systems must watch for indications that dashboard data does not represent actual performance metrics. Discrepancies should be addressed at the appropriate management level. This is especially true at the beginning, when the dashboard has to demonstrate its value to support an ongoing investment in maintenance and development.

Timelines

The need to comply with changing requirements within a limited time frame meant that development and implementation took place on a compressed schedule. The challenge is increased by the size and diversity of the organization and the decentralized command structure. Each of the challenges we faced was aggravated by aggressive timelines. However, the timelines were necessitated by circumstances.

As we discovered, organizations don't always have the latitude to progress along a measured time line. As management observed, the private-sector partner "is in a reactive mode to [the agency's] scheduling requirements, while [the agency] is in a reactive mode to [the demands of] OMB, [the Federal Information Security Management Act of 2002], and Congress." All experienced managers will recognize that implementing large-scale programs rapidly often results in a significant amount of rework.

CONCLUSIONS

1. The agency's CIO dashboard implementation, while extraordinarily complex, has not been inherently difficult. Complexity is what drives the development of CIO dashboards in the first place. Make sure you allocate sufficient time to manage the complexity, both at policy and technical levels.

2. The implementation was made more complex by the integration of strategic and tactical functionality. CIO dashboards were originally conceived as decision support software (DSS), designed to help top-level executives make strategic decisions for their organizations. While it is tempting to leverage the data in a DSS tool to make ground-level, tactical decisions, be cautious in deciding to "dual-purpose" a strategic tool. If you apply a strategic solution to tactical problems or use a CIO dashboard to evaluate tactical performance, the complexity of using such tools increases dramatically. Significantly, this usage will change the incentives of the mid-level management whose performance is being evaluated. Some of those evaluated may seek to increase their own reported performance by biasing the data they provide. In addition to lowering the reliability of tactical-level data in the dashboard, this will also lower the accuracy and value of the strategic-level data.

3. The development work for the CIO dashboard was significant, but much of the work is suitable for reuse. Each step in the definition of controls and supporting material contributes to development of a broadly applicable, commonly accepted set of controls, derived from FIPS-200, which will soon become a federal mandate. While participation in the process has been challenging, it has been a privilege to take part in the development of an extensible framework of controls derived from broad federal guidance. These controls can be shared across the federal government, and built upon across agencies, in order to meet the evolving federal mandates that impact information systems spanning the globe.

ABOUT THE AUTHOR

Lee Imrey is an IT Security Architect with a US federal agency, where he writes and refines policies to secure critical and classified information. He manages internal programs and works with other government organizations to implement practices consistent with those policies.

In his last job, he was Lead Instructor for (ISC)², the CISSP-certifying body. He taught classes internationally for private- and public-sector audiences. He was concurrently Senior Communications Manager and Lead Editor for the organization. In the latter role, he edited and produced the (ISC)² Newsletter, an electronic publication sent to tens of thousands of information security professionals worldwide.

Mr. Imrey has worked previously for telecommunications, retail, and consulting organizations, and as an independent consultant. He continues to contribute to the profession in volunteer capacities, speaking frequently at industry conferences. He serves as a member of the ASIS International IT Security Council, the (ISC)² Government Advisory Board, and is former Chair of the ISSA International Committee on Professional Ethics. Mr. Imrey is a Certified Information Systems Security Professional (CISSP), a Certified Information Systems Auditor (CISA), a Certified Protection Professional (CPP), and is board certified in security management through ASIS International. He earned his MBA at the University of Chicago Graduate School of Business. Mr. Imrey can be reached at cutter@imrey.com.