A STRATEGY FOR IMPLEMENTING BI/BPM TO GAIN COMPETITIVE ADVANTAGE

by Greg Mancuso and Al Moreno

INTRODUCTION

Today, enterprises are moving forward to integrate business intelligence (BI) and business performance management (BPM) applications into their infrastructure. There are a variety of reasons for this change, including gaining competitive advantage to pressures exerted by regulatory compliance from the US Sarbanes-Oxley Act (SOX), the Basel II Capital Accord, and the Health Insurance Portability and Accountability Act (HIPAA). Whatever the reasons driving your organization to implement these solutions, it's important to remember that deploying these technologies involves far more effort than drag, drop, and go.

While the benefits of these new technologies far exceed the negatives, organizations must be aware of the substantial effort required in the successful deployment of these tools and of the fact that these applications may place stringent requirements on a company's infrastructure. Such pressures can cause great pain to the IT organization as well as to the business side of the house. Coping successfully with this pain requires a tightly coupled partnership between IT groups and the business community. Both sides must recognize the limitations and boundaries within which they must operate.

Many enterprises today contain sophisticated levels of specialty applications designed to provide members of organizations with strategic pieces of information that allow quick reaction to industry activity and change. They afford groups a highly focused view of key business components and vital information by filtering out less pertinent information. When combined with integrated and refined planning techniques as well as revamped business processes, these tools allow for near-real-time monitoring of almost every aspect of an organization.

To date, the struggle has been to grow the ancillary systems and tools so that organizations can better integrate the various sources of information within their data centers and in users' heads. This allows for an increased level of detail and a more complete picture of the business situation at any given time. This picture can be monitored and compared with the company's budgeting, planning, and forecasting processes, as well as its strategic goal setting. This ability to plan and monitor began as a primitive, highly manual, and one-cycle pass process. Over time and through a massive organizational effort, it has evolved into a much more iterative and interactive set of activities. This evolution has mandated a new, highly evolved set of business and technical processes that are intertwined with and complementary to one another to provide instantaneous access to the corporate world of "information."

This report addresses this maturation process, from its evolutionary beginnings to what we foresee as future trends. Figure 1 depicts the typical key components and environment that results from the evolutionary process of an IT organization. Users become dependent on technologies and draw information from a variety of systems into their own workstations. Here, through either the use of Excel or Access or other similar technologies, users find ways to merge the data they need to produce the complex reports of today's organizations.

Figure 1 -- Corporate data silos.

GROWTH OF THE CORPORATE INFRASTRUCTURE

Need for Organizational Infrastructure

Previously, organizations could work with disparate data at a leisurely pace and had the luxury of time to transform data into usable information. The dawn of e-commerce, overnight mergers and acquisitions, and shortened regulatory reporting periods no longer affords this luxury. Companies must have infrastructure in place that permits instantaneous access to highly refined pieces of data. Further, these infrastructures must have processes in place so that data can be automatically processed with sophisticated business rules and technologies, with minimal human intervention, into reliable and usable information.

It is not difficult to understand how the world of IT has undergone massive revamping over the past 10 to 15 years. As the level of growth and the capabilities of systems to process volumes of data have matured, so too has the dependency of organizations on automated analysis. Unfortunately, growth in tool capabilities has occurred much more rapidly than most organizations can manage. As a rule, IT organizations are slow to adopt new technologies and must enforce their existing change management processes to ensure stability of the data infrastructure and validity of their information. As a result, the typical business previously had a loose infrastructure that existed to support what was in place and that expanded as necessary to accommodate the new pieces of technology that were required to keep the latest additional operational systems growing. As long as an enterprise could survive with the information and level of integration these makeshift infrastructures provided, organizations could function. While things seemed more the product of a kludge than a concerted effort, as long as minimally acceptable levels of data flowed forth, organizations could live with what they had.

When the world of business entered the electronic age, these architecturally unsound infrastructures could no longer meet organizations' needs; organizations strove for competitive advantage using transformed data. IT was faced with the harsh realization that it had to maintain what it had put in place while simultaneously revamping the organization's increased demand for more detailed, timely, and sophisticated levels of integrated information. Many organizations awoke to the reality that their infrastructures couldn't support the new technological demands and that they were in the dire situation of having to increase efficiency and productivity without destroying the old operational systems -- in many cases, with inadequate funds, reduced personnel levels, and a demand for new and unfamiliar skill sets. Many organizations couldn't support the new requirements and were forced to start from square one while struggling with new demands, increased governmental business regulatory requirements, and new sophisticated BI/BPM tool suites that imposed rigid data requirements unsupportable by current infrastructures.

Business Versus IT Needs

Under such conditions, the business-IT relationship has grown increasingly at odds. In part, this tension stems from fundamentally different perspectives. IT is concerned with the installation, maintenance, and hardware and software implications of new demands. The business perspective, on the other hand, views IT as data keepers who simply need to plug in the new applications and turn on the faucet so that the information flows. IT must be responsive enough to provide the required data and services to business users. Meanwhile, the business community must be patient enough to give its technology support staff the time to build the requisite infrastructure to support the new tools in an orderly manner. If either side exerts pressure to rush the deployment or the data, it can lead to a haphazard installation that won't satisfy the operational needs of business users and the technical needs of IT.

IT organizations have also contended with massive changes to infrastructures at a time when an unfavorable business climate has curtailed funds and staffing levels. The struggle to bridge the chasm between ramping up the delivery of information and maintaining required daily operations has resulted in a full-time juggling act for many within IT management.

Unable to cope with the perceived inactivity and lack of responsiveness of IT, many users have turned to self-deployed departmental solutions, which typically solve only short-term tactical issues and immediate business needs. Unfortunately, these departmental solutions themselves often become invaluable assets to departments. Business departments will adapt and survive, often by increasing their reliance on the tools that make their lives easier and job responsibilities attainable. This also leads to various departmental applications growing to the point at which they become long-term maintenance headaches that exceed a department's capabilities. Eventually, these applications get moved under the responsibility of IT departments. This places further strain on IT, because departmental solutions can blossom into enterprise-wide critical decision-making applications that impose additional needs and requirements. The end result is a cycle with no apparent end: users ask IT for more; IT cannot respond on a timely basis; departmental users look for interim solutions; solutions become long-term applications and further weaken IT's ability to implement long-range enterprise solutions.

Aside from the obvious strain on organizational resources, following this line of IT application growth creates high costs for any organization. Often, different departments select different toolsets that have overlapping functionality, creating a nightmare not only from a cost-of-initial-acquisition perspective but also from a maintenance perspective, since many of these solutions require hardware, software, and maintenance skills not readily available in the organization.

Proactive IT

Putting an end to this cycle requires that an organization make some extremely difficult choices. Organizations must step back and take stock. The cost of doing nothing eventually forces a decision. So it's better to designate a time for extensive self-examination, which necessitates significant effort and commitment. For organizations that have successfully moved past the described situation, the common thread is a willingness to take a hard look at infrastructure and to take the time to do some planning. Organizations have incorporated use of the many theories, such as the Corporate Information Factory by Bill Inmon, Claudia Imhoff, and Ryan Sousa. As depicted in Figure 2, if you look at the heart of such a strategy, it is commonly advocated that its structure not only be built on existing organizational needs but also address the inevitable events that take place in any large operational environment.

Figure 2 -- An information interchange strategy.

The partnership between IT and business is created by both sides sitting down to voice their concerns, issues, and responsibilities so that both groups can work successfully toward a common goal: creating applications that benefit the enterprise without overrunning budgets and irreparably damaging valuable relationships. By working together, both sides can accomplish their objectives, and the organization can benefit over the long term by having an infrastructure in place that not only services current needs but facilitates the assimilation of future growth into the enterprise. As the organization grows by either expansion or acquisition, it can add new applications and solve business problems with relative ease and in a short time frame. New applications don't have to be a source of constant panic and upheaval.

Foresight and Vision

Today, an organization's every IT move must be part of a master plan to support the organization. IT shops must have a firm handle on today's needs, a strong understanding of the business, and an uncanny ability to foresee the future. While previously it was acceptable to build infrastructures that would support the immediate needs of today, current needs may extend as far as three to five years, and the failure to take a longer-term view will probably create more issues than the time taken up front to plan. It is safe to assume that even if you don't have a current need for a given functionality, you will when another acquisition or merger forces the need. While it isn't feasible to identify and design for every possible scenario, today's IT organizational planners must have a vision and purpose that provides for easy incorporation of future technologies into the existing infrastructure.

Ten or 20 years ago, few businesses foresaw the onslaught of e-business and the need to satisfy a variety of internal and external user requirements. Organizations today provide valuable Internet and intranet functions that only a few years ago would not have been considered a business requirement. Those organizations that have suffered least in recent years had vision, exercised planning, and thought outside the box. Further, organizations often mistake planning for doing. Understanding that IT infrastructure must accommodate future growth doesn't translate into buying every piece of hardware or software to support those needs right now. A successful infrastructure can absorb requirements and incorporate the latest technologies because these events have been planned for. The advances in storage, memory, and technology in general facilitate this planning. It has become virtually acceptable for vendors to develop hardware with a built-in obsolescence factor. Newer and better equipment is now being produced on an annual basis, with platforms becoming obsolete in two or three years. Organizations that build their infrastructure without accounting for such rapid change will find their ability to stay current severely hampered. Maintaining a competitive edge necessitates that you understand what your competitors are doing and plan for similar functionality.

Building the Organizational Infrastructure

Clearly then, the task of creating the best environment to support the new wave of technologies must include a good deal of thought and planning. As business mandates new requirements, IT can respond in a timely fashion and avoid the perception of being unresponsive. IT managers today must plan in four key infrastructural areas. First and foremost, any infrastructure must be able to provide operational support for the existing day-to-day systems that drive the business. These systems include accounting, customer relationship management (CRM), enterprise resource planning (ERP), payroll, and HR systems, to name a few. Today, a variety of packages are written and supported for the most popular relational databases and function on common hardware platforms. IT support for these systems is now fairly commonplace and involves maintenance of the systems, backup-and-recovery processing, and general manpower to support the functional side of the business. Many IT shops do not understand that today's analysis tools draw from multiple systems of record and need tightly coupled and integrated information that has been highly cleansed and verified.

The second area of support and services involves providing data integration and allows for accumulation of large amounts of data from various systems. This environment typically contains data warehouses, data marts, and operational data stores. These repositories of data must be highly optimized for end-user queries and use, must be verifiable, and must permit the total audit of information and transformation back to the systems of record. This data can be tailored and customized, which then serves as the basis for the next phases of use. It is here that the individual data elements are married, matched, and altered from a set of discrete, disparate data to a coordinated and potent source of knowledge. Data maintained in this environment has been cleansed and matched in the transformation from disparate data into integrated information.

Third, the area of corporate reporting may also include a separate or complementary set of tools that permits the analysis of historical information, which can become a competitive knowledge base from which management can draw solid and verifiable information. In turn, this information forms the basis of management's strategic action plans. Because the data contained in the repositories is solid and verifiable, it is only logical that decisions based on this knowledge, when coupled with executive management's expertise and understanding of the business, yield valid and attainable strategic objectives that provide an enterprise with a strong competitive advantage.

Finally, the organization requires a means of disseminating information from the top down as well as from the bottom up. IT must have a firm handle on the corporate communications infrastructure, since requirements in this area may involve everything from intranet reporting to providing customers with transactional information about their purchases, complete with price analysis and historical trends. The communications infrastructure must be robust enough to handle volumes of Internet traffic, large amounts of electronic data movement, and a means of information exchange between systems and corporate entities. In general, the network can be your best friend or your worst enemy, and its role is dependent on bandwidth and the ability to expand with the growth of the enterprise. In short, this area, too, requires good planning and the ability to foresee future needs within the organization. The emphasis should be on planning for expansion, purchasing, and implementing only what is required for today while maintaining the ability to incorporate future needs into the existing infrastructure easily and with a minimum amount of pain.

BI/BPM Application Mandates

You might wonder why it is necessary to place such heavy emphasis on a strong, stable, and flexible infrastructure when moving into the BI and BPM arena. Companies that implement these applications gain significant competitive advantage, and the use of these toolsets creates a consistent ability to dramatically improve the accuracy of the objectives of executive management. These technologies thrive on the ability to have massive volumes of information flow from their systems, honed by BI/BPM applications, to provide near-real-time analysis and monitoring of conditions as compared with stated goals and objectives.

Traditionally, organizations created organizational direction through a highly time-consuming and manpower-intensive process. The process further required that strategic direction be set annually and the ability to provide feedback on some fixed basis, usually monthly. With the advent of today's BI/BPM tools, it's now possible to provide massive amounts of consolidated, cleansed, and coordinated information on an ongoing and continuous basis. This information is more timely than previously and is sufficiently easy to use, so even the most computer-illiterate manager can access the information in a format he or she can easily understand and manipulate. This ability allows constant oversight of the strategic direction while at the same time permitting and empowering management to make midcourse corrections with confidence. The tools' capability to provide detailed information from multiple points of view allows decision makers to readily make the changes required to drive new business opportunities and to retune every level of the organization.

In addition, these new toolsets and applications allow both strategic and tactical information to be directed to the organizational levels where this information can do the most good. Using Internet and communications infrastructure, field personnel can receive daily results of these efforts, permitting them to identify issues and areas of concern long before problems arise. All such capabilities, of course, are contingent on the ability of the infrastructure to support organizational needs to present the information in a manner and at a pace that serves the needs of the entire organization.

Corporate Data Structures

When trying to understand the totality of corporate data structures, one must account for the fact that several layers of data flow through an organization. Three key components of corporate data structures must be accounted for. The first is the operational data structure. This consists of structured data environments and unstructured data environments. Figure 3 illustrates the key components of these two worlds of information.

Figure 3 -- Corporate data structures.

This layer of corporate data comprises the world of hard-and-fast facts, figures, and dollar amounts. To date, organizations have spent almost 100% of their resources tending to the structured portion of the enterprise. Operational systems have been honed and carefully orchestrated to deliver high-capacity storage and retrieval of this key corporate data. Customer names, account histories, and historical financial dealings have all been captured, categorized, and optimized for analysis.

Oddly enough, the new regulatory requirements of SOX and Basel II have initiated a new trend in the business world. SOX and Basel II now require much closer monitoring of unstructured information consisting of paper documents, e-mail messages, and corporate communication. Organizations now recognize that there are few software options to deal with the need to merge the structured and unstructured worlds of data. Organizations capture orders and financial dealings, but few have ever contended with the volume of customer interactions that now occur electronically through e-mail or postal mail; nor is the massive amount of paper generated daily collected or captured. The information contained in letters, e-mail messages, and other corporate communications is now considered as important as the information contained in the operational world.

Existing back-office systems capture some information, but few, if any, tools are available to categorize and merge the information captured with structured information, resulting in the loss of significant information that could clarify and improve decision making. For example, the reason for the fact that some customer behaviors don't conform to corporate expectations may well lie buried in these volumes of untapped data. This is evident if you consider the volumes of unstructured information. Industry experts such as Inmon set the ratio of structured to unstructured at roughly 20%-80%. By extension, 100% of companies' decisions and goal setting is based on only 20% of the available information (a variation on Pareto's Principle).

The fact that organizations make decisions using only small amounts of structured information is further complicated by the lack of analytical tools to support the use of unstructured information. Most analytical tools today address structured information. The current crop of analytical tools uses facts, figures, and digitized information to look for trends, report on collected data, and provide BI/BPM applications that dissect and manipulate the available information. While previously these capabilities sufficed, the trend is a move to grab, categorize, and make use of unstructured data and, equally important, to use the unstructured data in the context of the current structured analysis.

This seems to be the wave of the future as enterprises struggle to move forward in search of competitive advantage. Spinning spreadsheets will no longer provide the benefits required to maintain industry leadership. Corporate competitive advantage is now largely contained in the unstructured information of e-mail and hard-copy communication. The difference between corporate excellence and failure has diminished so significantly that organizations that make the transition to incorporate this unstructured information will gain the definitive competitive advantage.

The final layer of corporate data, metadata, is often totally overlooked or underdefined. Metadata means different things to different members of an organization. In the traditional sense, metadata is informational data about the data that resides in your corporate systems. There are also multiple types of metadata, such as technical metadata. This information refers to the data elements that make up your IT systems. The name of the data elements, the length of the data, the type of the data, and so forth are all characteristics of your metadata. Examples would be the data element name CALC_PCT, calculated and stored in the metadata repository as real (or float, or floating point). Information may also contain the database table names in which the element is stored.

By contrast, there is also business metadata. Business metadata stores the reference information that makes sense to end users. This data may include the common name for a given element, such as margin percent, the reports it is used on, the system it is sourced from, and an English definition of the calculation formula. The audience for this data includes those looking at a saved calculation and wanting to know the underlying data elements used to create the cell in question.

Most enterprises either totally ignore the business metadata requirement or only populate the most basic information. Previously, this was acceptable, but as BI/BPM toolsets have taken hold, the business model and users have changed; users no longer look solely at report output because not only IT programmers but also business users work with these tools. Suddenly it has become imperative for organizations to understand the data that was stored from a technical perspective as well as an end-user perspective. This information must involve more than the cryptic database column names as understood by IT staff, because regulatory mandates force end users to be aware of the information they use and assign blame to those creating informational outlets for the company. In addition, users have added a new layer of "metadata" in the appearance of reporting structures.

Reporting is more than just formatting columns and rows. Specifically, it is a method of visually displaying the progress of the company for everyone to see. All reporting involves creating report levels, or hierarchies. Hierarchy management is a way of ensuring that all members of an organization use a consistent method to create reports. Hierarchy management now plays a vital role because SOX mandates a clear, concise, and repeatable reporting format. Hierarchies are logical rollups that can be disseminated to all the systems used for reporting and data analysis. By ensuring that hierarchies are maintained and managed in a central repository, it is safe to assume that everyone in the organization shares a consistent view of the business -- that is, of course, if everyone uses the same reporting structures. Many new products are emerging to perform this vital and often-overlooked functionality. Figure 4 depicts the relationships related to reporting hierarchies.

Figure 4 -- Organizational hierarchy management.

BRIDGING THE GAP: OPERATIONAL AND ANALYTICAL INFRASTRUCTURES

Business information needs vary greatly from user to user. Many corporate technology infrastructure users reside mainly in the tactical space; that is, they are business users with the sole responsibility of performing data entry for companies' various operational systems, such as order entry, G/L, ERP, and CRM applications. These users are often supported with targeted report output from these same applications that satisfy the various QA activities associated with their job responsibilities. Questions such as "Did I enter that order correctly?" and "What are the phone numbers for this customer group so that I can call and verify their information?" are common. As one moves up the hierarchy of the corporate structure, the need for more strategic or analytical data increases. Once those at the C level are using this data, questions rarely, if ever, concern the operational. C-level information needs center on corporate financial performance displayed and analyzed from multiple points of view or with inventory management and optimization based on a varying number of criteria.

Both of these disparate informational needs must be satisfied by the company's technology infrastructure and various online and offline data repositories. Even more important than the business community's lack of understanding about the distinctions between operational and strategic informational needs is the lack of understanding between the business community and IT support staff. The gap can never be fully bridged unless all parties take the time to jointly assess the current state of the company's strategic direction, business processes, and technology support structure.

In order to successfully bridge the gap between these two islands of information requirements, it is imperative that one has a complete understanding of the company's corporate information systems and that the organization performs a thorough analysis of the current and planned information systems.

Corporate Information Systems

Every company comprises many silos of data. Many of these silos house similar, if not completely redundant, data. Most commonly, each information system was designed and implemented to support a specific tactical or operational need. Companies build or deploy order entry systems to place and manage their customer orders and requests. Companies build or deploy customer service systems to support their existing customer base with new products, services, or other contact needs. Companies build or deploy sales automation systems to support their sales force and/or maintain their potential customer information. Alternatively, companies may deploy CRM solutions to satisfy many or all of these needs. Companies must also manage their financial and regulatory offices by building or deploying any combination of G/L, accounting, ERP, HR, inventory management, supply chain, material requirement planning (MRP), or other custom-developed tools and systems.

It is obvious how data collection and storage mechanisms can grow quickly. These tools and systems all require similar core data to function while also gathering significantly broad, diverse, and profuse amounts of other pieces of data. Even with their intrinsic ability to collect and display huge amounts of data, each of these types of applications is built to run day-to-day organizational operations. Sales cannot be fulfilled without functioning order entry, prospect management/CRM, or inventory applications. Factory output cannot be accurately projected and staffed without functioning order entry, inventory, supply chain, or MRP systems. Payroll and operating budgets cannot be projected and/or processed without functioning HR, G/L, accounting, or ERP applications. In short, a company requires each of these applications to open its doors in the morning.

On the other hand, management does not run a business in the same way that the business operates. Executives must look at information from several of the disparate operational systems and derive a view of the company's state in order to build strategic plans. To accomplish this, companies deploy analytic applications, such as data warehouses, BI reporting tools, OLAP cubes, dashboards, and scorecards. Just as these applications are used to define and drive the company's strategic direction, the analytic applications require significantly different design and implementation tactics than do operational systems.

Analysis of Corporate Information Systems

Regardless of the purpose of the information application -- operational or analytical -- no system can live in a vacuum. While the operational systems are designed to operate solely to support limited sets of critical day-to-day corporate functions, they routinely need to share key pieces of data. Also, operational systems will feed analytical applications. In order to accomplish this sharing of information, the organization should perform a complete analysis of all its information systems. In addition, the strategic analysis of the organization's data infrastructure should include an identification and assessment of potential future requirements. For example, the company does not currently use any CRM application; however, growth in business has necessitated the systematization of sales and prospect management activities. In doing this assessment, a strategic plan of the organization's current and future information needs is developed. This plan will also include a high-level requirements identification and assessment of any known future requirements. This plan becomes a roadmap that may be used when deploying new applications in the organization or when upgrading or enhancing existing portions of the infrastructure.

The Strategic Plan

As we noted previously, the strategic planning process involves taking a close look at all existing pieces of the corporate data infrastructure as well as identifying and prioritizing potential future needs. When planning to deploy any new information resource or to add components to an existing solution, time should always be set aside to ensure proper planning. The strategic planning efforts will help ensure smooth deployment and transition to new capabilities. The planning effort leads naturally to the development of a consistent integration architecture and process. It also ensures minimal redundancy in an effort to achieve the end result for the organization. As with all information initiatives, the process is iterative, and in some instances, portions of a current process are reworked in later phases. The up-front identification of these redundant efforts provides integration planners with the ability to better group requirements to deploy the tasks without reinventing the wheel during every release.

The strategic plan should include the following components:

• A technology architecture review
  
  
• An assessment of all existing systems and applications
  
  
• An identification and description of the desired future state of applications and infrastructure (including business requirements and hardware upgrades)
  
  

Technology Architecture Review

The current-state architecture assessment comprises several components:

• A graphic rendering of current-state topology components (a topology map)
  
  
• A topology map of planned future-state topology components
  
  
• A detailed definition of all systems platforms
  
  

Current and Future Topology Maps

Figures 5 and 6 show samples of current- and future-state topology maps. A topology map displays all current technology components and how they are connected to one another within the technology centers. Note that these maps do not identify interoperability or application-level dependencies. This information is included in a subsequent section of the strategic plan.

Figure 5 -- A sample current-state topology map.

Figure 6 -- A sample future-state topology map.

The topology map depicts all servers, client machines, network devices, and communications or network links. The hardware should be identified by its system name and, space permitting, a brief summary of hardware, operating system, and database engine. For example, a server in the topology map could be identified as "ServerName: Sun v440, Sol8, Oracle 9i," rather than simply as "Server."

Communications links should specify their bandwidth capacities and, if possible, utilization percentages. This information is useful when planning for any capacity increases necessitated by new applications to be developed.

Platform Definitions

Every component in a topology map should be completely defined. This definition includes the hardware profile, the operating system level (with all patches identified) complete software profile, and user or process load information. For example, the order entry application server may be defined as follows:

• ServerName: ORDENTSrv
  
  
• Hardware: Sun v440
  
  
• 4 x UltraSPARC IIIi
  
  
• 8 GB RAM
  
  
• 150 GB disk
  
  
• OS: Solaris 8, Revision 7/03
  
  
• Software: Oracle 9i
  
  
• Sun Java Application Server Platform 8
  
  
• User profile: one database administrator/system administrator
  
  
• Three power users/app server administrators
  
  
• 16 sales users
  
  
• Process load: system is generally 40% utilized at any time
  
  
• Peak utilization times are at the start and end of the business workweek and rarely exceed 60% utilization
  
  

Application and System Assessment

Key to the planning effort is the assessment of the application and systems. This assessment also tends to be the most time-consuming. Many organizations, especially those with larger and more established IT shops, run many applications and systems. Some of these applications have been in production for years, and the number of IT personnel who fully understand all parts of the application has decreased over time. Therefore, extra time and care should be paid to ensure that the applications are fully documented. The application assessment also includes a mapping of the data interdependencies between all systems and applications. Further, if they can be identified, all offline sources of information, such as Excel spreadsheets, should be incorporated into the application and system assessment.

The application assessment includes the following components:

• A data interdependency map (i.e., data flow diagram)
  
  
• A data interdependency analysis
  
  
• An application summary
  
  

Data Interdependency Map

All corporate technology assets inevitably have interdependencies. As the order processing application receives information about a new customer, the servicing application must acquire this same information to ensure that the service representatives have the appropriate knowledge to deal with any call for support. In order to ensure that all dependencies are documented and understood, the strategic plan should include a data interdependency map, otherwise known as a data flow diagram. Figure 7 shows a sample data interdependency map.

Figure 7 -- A sample data interdependency map.

Application Summary

The final piece of the application and system summary is the application summary. This summary takes an in-depth look at each source of data that the organization currently uses. Historically, these data sources were the operational "systems of record." More recently, the need to incorporate in a structured and systematic manner the various "personal silos" of data has presented itself. These silos include Excel spreadsheets, Access databases, or other information stored in offline resources. These personal silos should be considered when performing the application summary even if they are not currently used to feed data directly into an operational system. For example, the spreadsheet that the order entry clerk uses to manually look up valid values when entering data into the system is a vital piece of knowledge that should be captured at this time. Just because the system does not hold the decode map in its database does not mean that the information has no value.

This analysis includes the following items:

• The application name
  
  
• A description
  
  
• The hardware platform
  
  
• The software platform
  
  
• Business requirements
  
  
• A high-level data model
  
  
• A high-level data dictionary
  
  
• Hierarchy information
  
  
• Technical implementation parameters
  
  

Table 1 clarifies the contents and purpose of each of these items.

Table 1 -- An Application Summary

At the conclusion of these business and technology reviews, the foundation layers that bridge the gap of understanding between the business and technology communities have been established. By using these analysis documents, the organization is poised to build on a solid foundational understanding to drive the implementation of appropriate and abundantly useful technology to support the dynamic nature of today's business climate.

THE INTRODUCTION OF AN ANALYTICAL INFRASTRUCTURE

Companies of varying sizes have attempted to support analytical procedures and applications internally, which has led to many implementation approaches. All but a few have been plagued by false starts, missteps, or outright failures. The nature of analytical applications requires planning and implementation measures and techniques that are significantly more involved, time-intensive, and complex than the relatively simple deployment of an inhouse-developed business application. At a minimum, the analytical infrastructure comprises all components that support the strategic decision-making needs of the organization. These components include, but are not limited to, the following:

• Enterprise data repositories, such as data warehouses and data marts
  
  
• Analytical applications (i.e., OLAP), such as Hyperion Essbase and Cognos PowerPlay
  
  
• Data mining and visualization tools, such as IBM Intelligent Miner, Visipoint, and Quadstone Decisionhouse
  
  
• Reporting tools, such as Business Objects and Hyperion Brio
  
  

As can be seen from the admittedly small list of analytical tools above, many players are emerging in a rapidly growing industry to support the analytical needs of the business workspace. When planning an organization's integration framework, each kind of analysis must be considered. Otherwise, the underlying design of the deployed platform is likely to be inadequate, which minimizes the data available for any number of these applications.

Every IT shop in the world has some level of integration activity in place. In many cases, the sum total of past experience is in making finite sets of data from one legacy application available for use by another. Unfortunately, people tend to remain within their comfort zone; that is, they rely on the integration techniques that they have already deployed to satisfy the requirements imposed by the analytical applications platforms. This is where the trouble begins.

So how does an organization plan for the analytical portion of the integration framework? First, gather a thorough understanding of the business requirements, which directs the need for certain analytical applications. And second, research each component of the analytical platform. Each application will likely impose some informational or structural requirements on the technology integration effort.

Understand Business Needs

Time must be set aside to gather, analyze, and understand business users' requirements in terms of reporting and analytical capabilities. All these requirements are logically clustered into like groups: reporting, visualization, mining, OLAP, and so on. These groups of requirements identify which application components are necessary to comprise the analytical platform. Most platform implementations will require only a limited subset of the possible analysis applications.

Figure 8 -- A sample top-level data model.

Another point to consider is the scope of the analytical platform required to support the business. Many companies will start with a small, limited analytical implementation. This will be installed as a departmental solution, consisting of one or two applications. In most of these limited projects, the intelligence application will be fed directly from data contained in spreadsheets, desktop databases, or small extracts from a single operational system, such as the G/L. While this satisfies the needs of the company today, it doesn't involve planning to consider the future needs of the growing business. As a result, the integration platform and mechanism put in place will likely not be scalable enough to grow alongside the company either.

The organization that takes the longer-term view of the analytical needs of the company will inevitably be better positioned to succeed even if the IT group initially deploys only a subset of the total platform. By taking the long-range view, the development group is able to identify all potential data interactions, all integration requirements, and integration infrastructure components up front. This allows the design of a coordinated development and deployment plan for the integration framework.

This long-term approach is also the only path to overall success in terms of consistent data integration capabilities. No organization can survive by deploying tactical departmental analysis and integration projects. The departmental approach leads to extension of the systems' stovepipes that the integration framework is designed to eliminate. Large and/or forward-thinking organizations will begin the design of their analytical platforms and, by extension, their integration framework with the business requirements to drive the technical implementation of the integration framework.

Understand the Analytical Platform

The previous section discussed how and why analytical platforms' business requirements should be used as part of the planning phases of the integration framework. Once the tools have been identified, the next step is the design of the data repository portions of the analytical platform. It is commonly understood that business requirements lead directly to the organizational data necessary to be included in the data warehouse. Further, the data modeler will use the interactions between the business requirements to design the most effective data structures within the data warehouse.

Once the data interactions are understood and the basic structures have been defined, it is necessary to take a closer look at the other components of the analytical platform. Many of the tools on the market strongly recommend that certain data structures be built into the data repository in order to ensure optimal performance. In some cases, these recommendations are stronger than others. During the initial design of the warehouse data model, these recommendations must be known should the organization opt to use a product that actually mandates specific structures for data aggregation or relationship management.

One last point to consider is that not all tools work and play well together. Selecting the best server, the best database, and the best analytical products without regard for their interaction will not yield the best results. Some products are parts of complete suites that work effectively only with other products in the suite. Further, the supporting data structures required for one product may conflict with those of another.

BUILDING AN INTEGRATION FRAMEWORK

As this report indicates, maintaining an infrastructure that enables IT to provide all the needs of an organization is not trivial. It is difficult for many organizations to build such structures initially, and it is difficult to create such structures and have the necessary bandwidth to allow them to grow to accommodate changing demands. The key is to understand that any IT organization can maintain "data"; storing transactions alone does not provide the critical capability of providing "information." Here is the key distinction: Data is the result of underlying transactional events and is the data record in its purest form, stored as an electronic transaction; information, on the other hand, is the ability to relate various pieces of data to create useable facts. Data about sales to a customer during a time frame is a data point. Taking a series of data points for a customer and trending sales over time yields information. Customer X may buy more of a product during the summer than during the winter. This information is useful; if the trend holds true for any group of customers, it can be used to drive production and delivery decisions.

By extension, because the typical enterprise has collections of data points in a variety of unrelated systems, IT can provide a mechanism to integrate the disparate information that drives the quality of the information. This array of disparate information is critical to integrate, because it provides the information needed to drive strategic business decisions. Organizations must work on creating this infrastructure and capability because it is the quality of the information that will determine any organization's ability to respond to key tactical and strategic questions. Figure 9 depicts the required framework for creating a successful integration infrastructure.

Figure 9 -- The integration framework.

The infrastructure enables support of an environment that makes accessible all the operational data and any required external data that must be stored. The integration layer must be dynamic, flexible, and extensible. A typical environment may contain a data warehouse, perhaps some data marts, a hierarchy management application, and a metadata repository. If provisioned and dynamically established, such an environment will interact easily with the reporting and compliance layer to produce required analysis cubes, executive information systems, planning information, regulatory reporting, statistical reporting, and information for setting corporate objectives. Today, all the emphasis has been on building these layers to accommodate the structured organizational data. In the future, it will be necessary to incorporate the unstructured data contained in an organization's back-office systems. Future needs for compliance with SOX, HIPAA, and Basel II will force this incorporation.

Once the requirements for the organization are understood, only then should you begin to prepare the integration framework. Designing the integration framework should follow much the same path as designing a data warehouse:

1. Gather and validate the business needs.

2. Identify potential integration methodologies (e.g., custom code; extraction, transformation, and loading [ETL]; enterprise application integration [EAI]; enterprise information integration [EII]).

3. Evaluate candidate tools.

4. Deploy the selected tools.

5. Provide access to the framework to the developers who will implement the data integration processes.

The nature of your corporate IT culture will play heavily into which methodology or methodologies are suitable for your organization. The organizations with many inhouse-developed legacy applications will most likely lean toward implementing any integration task through the use of custom code or possibly the use of ETL products. On the other end of the spectrum, those organizations that rely heavily on the use of newer, packaged applications have many more options available to them, such as the application interfaces of EAI or the virtual data repositories made possible with EII.

In order to effectively plan the integration environment, designers must have a solid understanding of the various methodologies. Table 2 features each major integration method as well as its pros and cons.

Table 2 -- Integration Methods

The key to creating a successful environment is selecting complementary toolsets that can be placed in a well-thought-out, well-planned infrastructure. Creating such a model takes time, diligence, and intensive planning and information gathering. Organizations try and skimp on gathering requirements, but failure to undergo a full-blown needs-and-organizational-requirements analysis is much like an expectant married couple, eagerly awaiting the birth of its firstborn, who responds to the need for better transportation by buying a two-seater sports car. Such a solution may work in the short term but will be quickly outgrown once the child arrives. Solutions that are shortsighted and prove incapable of growing with the organization become more of a headache to maintain and support than the benefits they may initially provide. Even in the best-laid-out environments, understanding the gap between what is in place and what is being put in place mandates that an organization have the ability to respond quickly and without hesitation. The pain of response will be directly proportional to the success of the initial requirements gathering process as well as to how good a job is done at ferreting out what is likely to come up in the near term (three to six months) and in the long term (one or more years).

CONCLUSION

Today, the business world is fast-paced and business climates change rapidly, so business leaders must have immediate access to key business factors and performance metrics. This requires the capability not only to use the BI/BPM tools to see information in executive information systems (EIS), but also to view the detailed transactional data upon which the EIS information is based. Oftentimes, only by having access to the underlying detail are trends readily visible. In order to react quickly, the information must be organized effectively and presented efficiently. And to provide strategic value, these metrics must be presented from the viewpoints of different groups within the business. In order to accomplish this, the underlying detail must also be available within the context of users' analysis.

With the advent of new regulatory requirements, the reliance on much newer and automated data analysis tools has taken a dominant position in business users' arsenal for providing vital information for regulatory and statutory reporting. The mandatory nature of regulatory compliance has demanded a higher level of accountability and the creation of a far more foolproof and repeatable set of business processes that, supported by the new BI/BPM applications, provide the required level of analysis and detail to support reporting.

New trends in the business world mandated by SOX strengthen the need for new sources of disparate data. Whereas businesses were making decisions and building infrastructures only to support the use and analysis of structured data, new requirements now force the business community to deal with the world of unstructured back-office data as well. Today, a business must be fully accountable for all communications from and within the organization, and this massive amount of data can no longer be ignored. Future trends will force the volume of unstructured data to be unlocked and merged so all the appropriate data is available to the entire enterprise. Management can no longer sit back and assume that regulations are followed; it must now guarantee in writing that this is the case. This new organizational requirement forces enhancements and growth to current infrastructures as IT struggles for ways to integrate the structured and unstructured worlds to meet these new demands.

Since it falls to the IT organization to satisfy management's need for this new level of information and to create a relative infrastructure with the capabilities to support these new tools, IT must move forward. In order for the business to grow and thrive, IT must succeed. In order for IT to succeed, technology architects must take a step back and be proactive rather than reactive. Increasingly, software vendors are introducing themselves directly to the business community, and their technologies include capabilities that allow business users to accomplish some of the "traditional" IT workload of integrating data. As a result, business users tend to think that if these tools facilitate their jobs, it should be simple for an IT group to support the new technology. But this isn't always the case, and IT must be prepared to indicate why. The requirements under the covers for these new applications may not necessarily play nice with the current organizational infrastructure, and the cost of marrying new technologies with antiquated infrastructures is extremely high.

Technology people are notoriously conservative when it comes to considering new ways of accomplishing their jobs. When ETL tools first hit the market, for example, it was business sponsors -- not the IT community -- who first supported IT efforts for the decision support projects that drove IT away from developing custom code and toward performing integration and using new products. It was a business decision that used time to market and reduced staffing requirements to push IT management to use this new technology. Over the past decade, the integration space has not rested on its laurels. Now IT faces demands to evaluate, deploy, and/or support any combination of legacy custom integration code, ETL processes, EAI, and EII services. Given the various direct-access tools that many of the BI/BPM tools provide, the need for a coordinated integration framework is the only commonsense approach for an organization's CIO.

IT management must be willing to accept some short-term criticism from business users in order to buy the necessary time to plan for the ultimate success of all. By taking the time to understand the nature of the business, the needs of users, the current and planned future technology architectures, and hardware and software technologies currently in use and in development, IT planners can design an infrastructure that meets all of today's needs while remaining flexible enough to support the anticipated, and unanticipated, growth of the company. This integration framework allows all data-generating and data-gathering points in the company -- regardless of their location or types of data they store -- to communicate, share, and evolve into an integrated information factory that drives rather than hinders corporate success.

ABOUT THE AUTHORS

Al Moreno and Greg Mancuso are the founders and principals of Sinecon, a BI consultancy specializing in data integration and BI/data warehouse solution architecture design. Prior to founding Sinecon in 2003, they were employed with Hyperion Solutions, a leading provider of BI/BPM solutions. They were responsible for the development and management of the company's data warehouse solutions and data integration services group. Their articles on BI, BPM, and data warehouse topics have been published in DM Review, and the authors are monthly columnists for DM Direct, DM Review's e-mail newsletter. They are regular speakers at industry conferences and have presented at seminars on BI and data warehouse topics in the US and Europe. Together, they have more than 31 years of data warehouse and BI experience and have implemented many large-scale solutions in the US, European, and Asia-Pacific markets. Mr. Moreno can be reached at amoreno@sinecon-llc.com, and Mr. Mancuso can be reached at gmancuso@sinecon-llc.com.