TDWI Clarifies the Big Data Opportunity & Challenges

Last week, I posted about the implications of “big data” on the future of BI, and why we’re skeptical about many of the bold promises of business benefits without adequate coverage of the hurdles involved. We’ve also spent time with TDWI’s Big Data Analytics research results (registration required), which offered a refreshingly pragmatic tone in all the recent noise about “big data.”

The findings were especially interesting because a good balance of IT pros and business stakeholders were surveyed. (We at Balanced Insight admit to user bias when it comes to BI; after all they own BI these days.) The report contains many surprising points, raises a number of critical issues, and outlines the current state of “big data.” Here’s what caught our eye:

Where Big Data Lives & How It’s Processed: TDWI clarifies how big data differs from standard business intelligence and data warehousing practices, in that

big data analytics explores granular details of business operations and customer interactions that seldom find their way into a data warehouse or standard report

Big data is often not “well-understood, auditable, and squeaky clean.” The report also made clear that leading-edge platforms are capable of processing huge amounts of data in record times (translation: it’s not about the technology). Organizationally, the BI/DW teams are the most common owner of “big data,” according to 41% of survey respondents.

What to Call Analysis of Big Data: “Advanced analytics” may not be the optimal name for what analysts will do with big data, according to survey respondents, who thought “exploratory analytics” or “discovery analytics” were more accurate. I like the emphasis here on interaction with the data, with users able to “explore” and “discover.” This will be key to success with big data analytics (or whatever it ends up being called).

In other words, with big data analytics, the user is typically a business analyst who is trying to discover new business facts that no one in the enterprise knew before.

To our ears, this sounds suspiciously like effective business intelligence, though perhaps “BI on steroids” would be a fitting term provided users could actually ask questions of the data. I couldn’t help but chuckle at the survey respondent who wanted to name this new type of analysis “we-need-to-buy-more-hardware analytics.”

Impact on BI & Challenges to Adoption: Significant percentages of survey respondents believed the impact of big data on BI would be mostly beneficial, thanks to:

more numerous and accurate business insights (45%), an understanding of business change (30%), better planning and forecasting (29%), and the identification of root causes of cost (29%)

All that sounds good, but compare these benefits to the barriers to adoption cited by survey respondents. Resource scarcity is the top concern, highlighted by 46%. The “skill set is not quite the same as that for business intelligence and data warehousing,” the report warns. “Lack of business sponsorship” was the third most frequently mentioned barrier at 38%, while “cannot make big data usable for end users” was cited by 22% of respondents.

It’s not hard to see how the last issue could become a much greater challenge as big data grows more prevalent. And while big data has additional technical challenges in this regard, I want to point out that few organizations have been successful making normal volumes of data usable for end users.

It seems that big data has a lot in common with BI and that much future overlap is likely. The business benefits are similarly compelling, and similar challenges present similar difficulties. Process and organizational factors (perhaps including development methodologies) will have a lot to say about ultimate big data success, just as they do with BI. It will be interesting to watch the evolution.

The bottom line is that big data is here, it’s real and it’s not going anywhere – even though the industry is still struggling to define it precisely or calculate its impacts. In the meantime, we’re thankful that TDWI has contributed useful research that will help the BI industry begin to sort the potential innovations and advancements from the considerable hype.