Big Data Fatigue?

It may be everywhere now, but big data matters more than ever.

Reading Time: 4 min 


Competing With Data & Analytics

How does data inform business processes, offerings, and engagement with customers? This research looks at trends in the use of analytics, the evolution of analytics strategy, optimal team composition, and new opportunities for data-driven innovation.
More in this series

I was having lunch recently with the editor of a leading business publication, when I was asked, “Don’t you see a backlash against ‘big data’? Aren’t people growing tired of hearing about it?”

This seemed a fair question, given the sheer volume of articles and media coverage on the topic of big data during the past three years. But my response was a resounding, “No!”

During the summer of 2013, NewVantage Partners conducted a survey of Fortune 1000 C-suite executives from companies including American Express, CVS Caremark, JP Morgan, Johnson & Johnson, Kaiser Permanente, MetLife, Travelers and Wells Fargo, among others.

Our survey found that 91% of executives indicated that they have a big data initiative planned or in progress, with 60% reporting having an initiative completed.

Let’s consider three reasons why big data is becoming part of the mainstream now.

1. Big data is about all data, not just social media, unstructured or massive data.

I’m sometimes told by senior corporate executives that they don’t have a big data need because they are not focusing on social media data, unstructured data or massive data sets. This is a common misconception about big data. While much of the talk about big data focuses on the benefits and opportunities that result from new sources of data — including social media, sensor and visual data — most of the action among mainstream corporations is focused on integrating information from traditional legacy environments, like COBOL and mainframe data sources.

Integration of legacy data remains one of the main challenges for most corporations born before the digital era. When asked what kinds of data corporations planned to integrate using big data capabilities, most respondents to our survey responded that their focus was on customer transaction and financial data. For corporations, the ultimate ability to link behavioral, transaction and customer interaction data provides insight into the relationships between what customers say and what they do.

Similarly, big data is not exclusively about massive data sets. It is often assumed that big data is exclusively about capturing very large volumes of data. But big data is also about integrating more sources (“variety”) of data, which needn’t be massive. As Babson College’s Tom Davenport states in a


Competing With Data & Analytics

How does data inform business processes, offerings, and engagement with customers? This research looks at trends in the use of analytics, the evolution of analytics strategy, optimal team composition, and new opportunities for data-driven innovation.
More in this series

More Like This

Add a comment

You must to post a comment.

First time here? Sign up for a free account: Comment on articles and get access to many more articles.

Comments (3)
carnot antonio romero
Good piece. Interesting, though, that you suggest that data preparation is less important in the age of big data... if anything I would expect quite the opposite. 

Granted, you don't do the classic data warehouse ETL, but every time you try to do something on Big Data you have to figure out how to transform it into something that exposes the information you really want to analyze-- filter out irrelevant records or fields, flatten it or structure it, pre-aggregate it... It's just that now you may need to do it over and over again for every project, what with 'schema on read' and all. (Of course, you only do enough of it to answer the question at hand rather than building the perfect data warehouse... so that's potentially a savings, and an opportunity for a shorter time to first answer vs. waiting on a big bang data warehouse that never finishes. At a minimum, it means spreading out of the costs of prep work across each project that consumes the data.)

We need to:
- simplify the transformation design work to where most business analysts can manage without IT
- help users figure out whether what they have is of adequate quality, i.e. fit-for-purpose, and how to get it there if it isn't already
- democratize the cataloging, curating and searching of the great big bags of bits
- automate the discovery of relevant data and relationships for the specific problem at hand
- enable the identification and re-use of data prep work from other projects where possible, without committing to pre-computing, materializing and constantly refreshing the outputs in case someone wants them

New practices will be developed, new tooling will facilitate all this and really trigger the trip up the Slope of Enlightenment. But in the meantime, many will slog, shoulder-deep, across the trough of disillusionment, a lucky few encouraged by a backhand's brush against some crystallized chunk sunk in the sump, hinting that a toe might soon find purchase on the Slope of Enlightenment... 

But enough of metaphor! Clearly I need sleep. See you all on the Plateau of Productivity...
Steven Gold
Thank you for your most insightful article! I would like to add some general observations of my own, gathered from work experience and academics in a variety of key functional areas. Please consider this comment as a type of “discussion draft” – here, the goal is to help facilitate discussions about what Big Data is and how we can best utilize it. Yes, “Big Data” is receiving a lot of attention from Senior Managers. Data Scientists, and IT Departments alike. In some ways, Big Data is the confluence of (1) the huge amounts of data being generated every day from new data sources which may not have existed 20 years ago, (2) the dramatic reduction in cost to store this “new” data provided by the Hadoop File System (HDFS), and (3) the development/availability of new technologies and Hadoop Ecosystem tools for storing, managing, and deriving actionable insights from the data. While these technological developments are both impressive and helpful – from my perspective, they may not always be self-sufficient in their own right. For example, from my perspective, some of the most important aspects of understanding and deriving actionable insights from Big Data are (partial list): (a) Metadata. To what degree do we know what the data means and what it does not mean? To what degree can we incorporate qualititative factors in our metadata? To what degree are we able to update and modify (sometimes selectively) metadata as the meaning and/or the structure of the data changes over time? (b) From my perspective, I view the domain knowledge of the Data Scientist as one of the keys to deriving more reliable data insights. For example, when analyzing Federal Reserve Bank economic data, a Data Scientist that understands the complex interrelationships between multi-variable data sets may be better able to provide value in this type of analysis. Here, from my perspective, it is not only the application of inferential statistics, machine learning, heuristics, etc. The Data Scientist, or Subject Matter Expert, may be able to provide perspective and insights, adding value which might otherwise possibly be overlooked. Hopefully, given sufficient interest, we can discuss this and more via future comments.

Very Respectfully,

Steven Gold

MS, Information and Telecommunications Systems, Johns Hopkins University, Baltimore MD

MBA - Marketing, University of Missouri, Kansas City MO

Strategic IT Security, LLC
Doug Laney
Couldn't agree more Randy. At Gartner we have compiled hundreds of real-world, high-value use cases of Big Data & advanced analytics, and we're learning of more everyday...particularly as you say Big Data projects are going "mainstream." People often misconstrue Gartner's Hype Cycle. That is, as a technology or technology concept becomes post-peak and starts sliding into the "Trough of Disillusionment" such as Big Data is now, this merely means that the fast-followers and laggards have more difficulty with it than the early adopters--mostly due to cultural, organizational and leadership barriers. Of course this is also when the usual boo-birds and naysayers come out of the woodwork.

Also, note that we have long contended that Big Data is about more than just "volume". My paper from 13 years ago first introduced the three-dimensional challenges and opportunities of increasing data volume, velocity and variety (the "3Vs").  Ref: And by 2:1 our clients tell us that data "variety" is the biggest challenge. 

Doug Laney, VP Research, Gartner, @doug_laney