Curt Hall
Senior Consultant

Curt Hall is a Senior Consultant with Cutter Consortium's Social Networking practice and Business Intelligence practice and author of its survey-based Executive Updates. He is the former Editor of Cutter's Business Intelligence Advisor and Intelligent Software Strategies newsletters and former associate editor of Cutter's Object-Oriented Strategies and Application Development Strategies newsletters.

Mr. Hall's expertise includes BI, data warehousing, data mining, and other analytical technologies and products. He also focuses on the commercial application of artificial intelligence including intelligent agents, expert systems, case-based reasoning, neural networks, speech recognition, and fuzzy logic. His study on the corporate use of data warehouses and enterprise analytic technologies has resulted in the in-depth Cutter Consortium report Corporate Use of Data Warehousing and Enterprise Analytic Technologies .

Mr. Hall is coauthor (with Paul Harmon) of Intelligent Software Systems Development: An IS Manager's Guide; a contributing author to James Martin and James Odell's Object-Oriented Methods: Pragmatic Considerations; and the author of Cutter's Data Warehousing for Business Intelligence . Mr. Hall's work has appeared in a variety of technical journals and IT publications. He can be reached at consulting@cutter.com.



Boosting Data Quality

An interview with Curt Hall, Senior Consultant, Cutter Consortium

A recent Cutter survey shined a light on data quality and data warehousing practices. In this interview, Cutter Senior Consultant Curt Hall addresses the importance of data quality, as well as the consolidation of data warehouses and data marts.

Q: In a recent Executive Update, you addressed the issue of data quality and how an overwhelming number of organizations are dissatisfied with the quality and integrity of their customer data. Why is that?

A: We've found that only about 15% of organizations are completely confident with the quality of their customer data. In other words, about 85% are making important decisions based on data whose quality they have reservations about. I know it's become a cliché to say that data quality is paramount to the success of any organization's data warehousing and BI efforts. However, data quality is important for other enterprise efforts, too. This is especially true for CRM. Moreover, I think many organizations are going to really begin to feel the impact of poor data quality as they ramp up their business performance management efforts. Simply put, the quality of the analyses generated by any application designed to measure business performance can only be as good as the data that are used to feed them. And if you're not confident in the quality of your data, there is always going to be that nagging feeling that your analyses may be weak.

Q: What efforts can organizations make in turning around that dissatisfaction?

A: Unfortunately, as I've said before, data quality efforts often represent a massive undertaking. You can't just buy some software, install it, and sit back and say, "Well, we've fixed our data quality problem." The reason that ensuring data quality is so difficult is because it requires making an ongoing commitment whose success ultimately comes down to people caring enough to see that the data is correct in the first place. This requires an attitude that must permeate throughout the ranks of the organization in order to get people to take responsibility for the quality of the data they generate. Instilling this responsibility is probably the most difficult part of any data quality effort, because it almost undoubtedly requires implementing data quality processes, tightly coupled with employee training. And for this to happen requires someone, or more likely some group, to take charge of the organization's overall data quality effort. This group's responsibilities should include: conducting an initial data quality audit to assess the integrity of the organization's data; cataloging the types of data quality practices already in use (and measuring their effectiveness in order to identify areas for improvement); and ensuring that employees are properly trained so they don't inadvertently circumvent the processes put in place, thereby enabling the organization to reach its data quality goals.

Q: Tell us about the importance of data stewards.

A: The importance of data stewards should not be underestimated. It is the data steward who is charged with the authority to see to it that the practices defined by the data quality group are implemented -- and enforced -- throughout the organization. Organizations with one or more designated data stewards will be in a much better position to successfully enact and enforce standards for data quality across their organizations than those that don't.

Q: You also recently addressed the issue of consolidation of data warehouses
and data marts. Why is that so important today?

A: A lot of organizations today are updating their BI capabilities to facilitate new analytic applications that can support new business management strategies. In particular, the current corporate fascination with the "two BPMs" -- business process management and business performance management -- has made it more important than ever that organizations obtain an integrated view of the enterprise.

But having multiple data warehouses and data marts scattered about different departments and divisions simply is not conducive to obtaining this integrated organizational view. Different departments and divisions typically employ different data management tools, metrics, and key business definitions, all of which can lead to confusing and non-standard analysis and reporting methods. Lack of standard architectures, practices, and metrics can also hamper organizational attempts to share analytical results and foster collaboration and feedback among decision makers, all of which are important for implementing new policies designed to measure and optimize business processes.

Consequently, it's no real surprise that forward-looking companies are seeking to add structure and standardization to their data management applications by consolidating multiple data warehouses and marts into an enterprise data integration architecture that can provide the practical capability to distribute analytics to all important facets of their organizations.



Business Intelligence Trends: Partner Relationship Management, Real-Time Data, and Wireless

An Interview with Curt Hall, Senior Consultant, Cutter Consortium

Q: What are the general trends in the business intelligence area?

Initially, people looked at data warehousing and business intelligence in terms of how they could improve sales using customer-related data. Now, companies are realizing they need to extend their analytical applications to encompass their supply chains as well. An enterprise-wide view takes into account not just customer related activities, but also your partners, your suppliers, possibly some of their customers, and your manufacturing operations. It is a lot bigger picture to deal with.

Companies are finding that by looking at production data, they can not only reduce their inventory across the supply chain, but also manage their partners; we're starting to hear the term "partner relationship management." A typical application of partner relationship management is using analytics to identify your top-performing suppliers and measure their performance over time. Based on those measurements, you can come up with a series of metrics and negotiate performance-based agreements.

On the front end, you might take complaint data, production data, and design data and mix that with information from call centers, help lines, and the help area of your Web site to identify products that may affect customer service levels. Intelligence gained from this can serve several purposes: it can be used immediately to alert your sales reps to various issues, allowing them to more proactively manage their customer relationships. Longer term, it can be used to correct flawed product design or service offerings that are negatively impacting customer satisfaction.

The other trend I'm seeing is toward real-time data warehousing. When data warehousing was new, the idea was to extract the data from various customer information systems, move it into the data warehouse, and run reports from that to try to figure out a company's best customers, why some customers were leaving, etc. An analyst would generate reports, management would look at it, a week or two later they would make a decision, and new directives would go out. Real-time data warehousing (also called active data warehousing) is automating this process, with all of these functions taking place in the warehouse. The analyses are instantaneous.

The airlines are a good example of this; they use real-time data to sell seats -- the closer you get to flight time, the more expensive the seats are. It's also being used for tracking and scheduling trucks. If a certain truck is late, the shipping company needs to know what's on it, down to the package level. They want to be able to notify customers that their packages are going to be late. If it's something that's not time sensitive, they can simply offer the customer a small refund. But if it's medicine or pay checks, they need to know so they can reroute the packages. These are applications that require a data warehouse. Some people are saying you don't really need a data warehouse anymore -- that you can write a specific application for these types of functions -- but that's not going to give you the flexibility you need. To have flexibility in your business intelligence, you need to have a data warehouse and build the functionality you need on top of it.

Q: How are companies getting these systems to work across the enterprise? Is it proving difficult?

It has become almost a cliché to say that you need to integrate your existing systems into any e-business environment, but it's true. And companies are certainly struggling with this. For example, if you go to an e-commerce site and order a book, you'll get a message saying something like, "This book usually ships within two days." What they're really saying is that if they have the book, they will ship it within two days. If they don't have it, you'll get a message in a day or so saying "That book is currently out of stock. May we recommend [these books]." What's going on is that there's no integration between the e-commerce system and the core transaction processing environments.

You can get away with this if your company is selling books or CDs. But companies that want to integrate their partners into their supply chain electronically and tie that into their data warehouse and other analytical operations are facing a major undertaking. Some companies are building everything themselves; others are going with an ERP system like SAP or PeopleSoft. The ERP vendors are enhancing their systems to support this type of integration. They have had to because, despite the downturn in the market, companies are continuing to build e-business applications. The combination of ERP and BI systems is the focus of the report I'm currently writing for the Business Intelligence Advisory Service.

Q: What about the recent mergers and acquisitions in the BI market?

There is a lot of movement in the BI market right now, including some interesting mergers and acquisitions. One of the most interesting is IBM's purchase of Informix. A lot of people have said that IBM purchased Informix just for their customer list, and that was certainly part of it, but there's also a technology factor. IBM now has Red Brick warehouse, which is a terrific program. Informix also owned about ten databases from their various acquisitions, some of them embedded databases and some that do time-series data analysis, which I'm sure IBM is going to bundle in their DB2 database.

There was also an interesting merger between Broadbase and Kana. Kana was a large e-business platform provider; Broadbase was an analytical CRM provider and had recently acquired some operational CRM technology. Both companies were in a precarious position (low sales and not much cash on hand); together, they should be able to provide a more complete e-business environment that can not only support the transaction processing and content management sides of e-business, but also data integration, analytics, personalization, and CRM capabilities (both operational and analytical), which are a necessary requirement for serious e-business applications.

Q: What's going on in the wireless BI arena?

Most of the wireless focus so far has been on providing data at the consumer level -- offering an ad on someone's cell phones as they walk past Macy's or whatever. But companies are now beginning to provide wireless BI capabilities to people in the field, and the number of wireless vendors selling these types of applications has doubled in the past six months. Most of the applications are designed to provide sales people or technicians with access to BI through their wireless devices. For example, if a sales person is going to call on a client, they can first use a prebuilt query to check on the status of all the client's current orders. That way, instead of walking in to sell and being blindsided by "What about this order that's been delayed?" the sales rep is forewarned and can be ready with an answer ("I know you're waiting on X, and that's being shipped as we talk. In the meantime, I'd like to tell you about a special promotion...). In addition, wireless BI devices provide the ability to write-back to the database or CRM systems, so technicians can update their accounts, etc. This is a very important feature.

I have also seen some great wireless CRM applications that are in beta right now. Say an executive is traveling and receives an alert that her division is not going to meet their target for this quarter. With a wireless device, the executive can run queries to get details on why the target was missed, without even having to find a place to plug in her laptop. She can immediately start writing her report or walk into her next meeting with detailed information on which regions or sales people missed their goals.

The BI vendors are well positioned to get in on the wireless services market, because their tools are already good at slicing and dicing data, whether its operational, production systems, or customer data. The software can just as easily send an alert saying "We're getting a lot of customer support calls regarding the version X release of our software" as "We're running into X, Y, and Z on the production line" or "Inventory is stocking up in this location." Even more importantly, business intelligence products have robust personalization capabilities, which is key with wireless. You have to target the right content to the right person on the right platform. The small screen sizes of these devices mean that users need to be able to completely customize the application to their needs. The technicians in the field wants to use their Palm Pilots to get daily reports, the executives want to be able to get information in text format on their cell phones, and so on. The BI vendors are well positioned for this technology-wise, whether they can pull it off marketing-wise remains to be seen.

Curt Hall