A CIO’s strategy for rethinking “messy BI”

Image: A CIO’s strategy for rethinking “messy BI”Take the initial steps toward your next-generation data architecture.

Download Technology Forecast: Spring 09.

As CIO, you know you have an information problem. You’ve spent countless dollars and staff hours getting your data warehouse, financial systems, customer systems, and other transaction systems to generate meaningful reports. You’ve led Herculean efforts to regularize, transform, and load that data into consistent formats that business intelligence (BI), enterprise resource planning (ERP), analysis, reporting, dashboard, and content management tools can handle. Yet company executives keep asking for more detailed information to make better decisions, especially about the emerging challenges in the ever-changing markets the company is trying to navigate.

There’s no way traditional information systems can handle all the sources, many of which are structured differently or not structured at all.

The reason for this state of affairs is not that BI and related systems are bad, but that they were designed for only a small part of the information needs businesses have today. The data structures in typical enterprise tools—such as those from IBM Cognos, Informatica, Oracle, SAP, and SAP BusinessObjects —are very good for what they do. But they weren’t intended to meet an increasingly common need: to reuse the data in combination with other internal and external information. Business users seek mashup capabilities because they derive insights from such explorations and analyses that internal, purpose-driven systems were never designed to achieve. PwC calls this “messy BI.”

People have always engaged in informal explorations—gleaning insights from spreadsheets, trade publications, and conversations with colleagues—but the rise of the Internet and local intranets has made information available from so many sources that the exploration now possible is of a new order of richness and complexity. Call it the Google effect: People expect to be able to find rich stores of information to help test ideas, do what-if analyses, and get a sense of where their markets may be moving.

There’s no way traditional information systems can handle all the sources, many of which are structured differently or not structured at all. And because the utility of any source changes over time, even if you could integrate all the data you thought were useful into your analytics systems, there would be many you didn’t identify that users would want. You don’t want to create a haystack just because someone might want a specific straw at some point.

Tom Flanagan, CIO of Amgen, a biomedical company, sums up the problem: “It is difficult to get the business to very accurately portray what its real requirements are. With the type of business intelligence that we have, almost invariably we end up having to build these data cubes, and we build them based on the requirements the business gives us. What we build oftentimes does not meet the business expectations. It may meet what they said they wanted, but in actuality they want a very flexible type of reporting that gives them the ability to drill down to whatever layer of detail they want. So, the challenge with the historic way of providing reports is that it does not meet that flexibility that the business demands.”