Big Data & the Future of BI

2011 may be remembered as the year of “big data” in technology circles. Certainly, that’s true in business intelligence and analytics (broadly defined). IBM’s Watson was part of the story, as was the report from McKinsey Global Institute that came out a few months ago.

Titled “Big Data: the Next Frontier for Innovation, Competition and Productivity,” the report was full of fun facts (30 billion pieces of content are shared on Facebook every month and all the world’s music could be stored on a $600 disk drive) and bold claims. For instance,

The use of big data will underpin new waves of productivity growth and consumer surplus. For example, we estimate that a retailer using big data to the full has the potential to increase its operating margin by more than 60 percent.

Elsewhere, the authors described annual savings of $300 billion in the U.S. healthcare market from the effective usage of big data.

Those eye-popping numbers raised a few skeptical eyebrows (mine included). Sure, retailers have come a long way in using data to make targeted offers to consumers, but even advanced users – like Amazon – still have fairly primitive recommendation engines. Obviously, there is great potential value from huge data, but harvesting it will require the same tools, skills and processes most companies are still searching for in order to use plain ol’ business intelligence “to the full.”

Further, the authors never quite specify what “using big data to the full” really means. In our view, it would entail enabling business analysts and users with easy-to-use tools that allow them to interact with core data, creatively and interactively. That means posing lots of “what if” questions and probing even deeper based on the initial answers the data provides. Sounds like highly effective BI to us.

Gartner’s take is that the volume and storage issues are a red herring, because:

While big data is a significant issue, … the more important one is making sense of big data and finding patterns in it that help organizations make better business decisions.

In other words, the high hurdles of using big data “to the full” are the same high hurdles of using normal volumes of data “to the full.” So, if these big dreams about 60% operating margin increases from big data are to come true, effective BI principles are going to be a big part of the story.

If you list the top five reasons that companies have not used BI “to the full,” nowhere in that list will you find “too much data to process.” That problem only exists with unstructured data (e.g., when the web is a data source). So how does the advent of parallel data processing solve the problems that BI has been struggling with for years? A very good argument can be made that big data is about to make some of BI’s core challenges worse, since parallel data processing requires a far lower-level programming language to work with as opposed to BI.

Maybe McKinsey’s next report will be about “agile data.” In the meantime, we expect we’ll be breaking down the idea and implications of big data in future posts.