APPLYING AGILE PRACTICES TO DATA WAREHOUSING

by Dr. Ken Collier, Senior Consultant, Cutter Consortium; with Jim Highsmith, Director, Cutter Agile Project Management Practice

Traditional data warehouse projects follow a typical waterfall development model in which rigorous efforts are made to collect complete requirements, design comprehensive architectures and data models, develop and populate repositories, and, ultimately, develop the analytical reports and artifacts that users want. These projects are complex affairs, involving various stakeholders. Depending on their magnitude, these projects generally run at least six months and can easily exceed US $1 million.

But often, despite the best efforts of developers, project managers, and stakeholders, projects run over budget and past deadline, and business users are less than ecstatic about the first data warehouse production rollout. For many data warehousing project failures, the root cause is the disconnect between user and developer expectations.

Our experience suggests that an agile data warehousing (ADW) approach greatly increases the likelihood of successful implementation of a data warehouse on time and within budget.

ADW meshes well with existing data warehousing architectures and practices. The following practices have proved most effective:

Practice 1: frequent, short iterations. Each iteration produces a working and demonstrable data warehouse. Although the length of iterations varies, the aim is two-week iteration cycles. This time frame is long enough to build something meaningful and to undo mistakes without it being too costly. Frequent and short iterations accomplish several goals: (1) users are directly involved in the development cycle; (2) users and stakeholders can see the real progress of development; and (3) developers can verify that they are on the right track.

Practice 2: the conceptual phase. This iteration establishes the entire infrastructure necessary for sustainable agile development.

Practice 3: develop features, not functions. Agile development is driven by the completion of customer features rather than functions. The aim is to produce demonstrable features that users can review and evaluate in light of business requirements.

Practice 4: develop production-quality features. Production teams should create production-ready features; users will be in a better position to provide valuable feedback.

Practice 5: Test-Driven Development (TDD). Requirements determine the tests: as soon as your code passes the test, you have met the requirements related to the test.

Practice 6: automated testing. Typical system testing is often handled by a dedicated QA team near the end of the project; this is not well suited to the agile practice of testing during development. An agile automated approach allows for testing each table in the staging database and in the warehouse repository as well as the validation of data in the final reports and models.

Practice 7: incremental design. Like testing, design and modeling should be integrated into the agile development iterations. While conceptual design largely occurs during the envision cycle or during iteration zero, ADW practitioners should not shirk the responsibility of detailed design.

Practice 8: barely sufficient modeling. While modeling is critical to success, effective and agile modeling requires producing only those models necessary to achieve the intended goal of either understanding or communicating. Agile models should accomplish but not exceed their intended goals.

Practice 9: build the right project team. No formalized development process can substitute for the right people; and people are not plug-and-play resources.

ADW creates an environment that promotes continuous innovation by satisfying users' requirements. It also promotes system adaptability through the ability to rapidly respond to change and to accommodate future user requirements. In 2002, IEEE Software reported that agile methods demonstrated significant improvements in productivity, cost, and cycle time relative to industry benchmarks. Although informal, initial experiences in ADW appear to support these benefits. By adopting an agile model of early and incremental design and testing, developers can produce high-quality data warehouses and ensure that users' needs are met effectively, that the users' expectations are realistic and appropriate, and that developers can adapt to evolving user requirements. ADW helps deliver the system according to changing requirements, on time and within budget.

-- Dr. Ken Collier, Senior Consultant, Cutter Consortium; with Jim Highsmith, Director, Cutter Agile Project Management Practice