Big Data Fatigue?

Good piece. Interesting, though, that you suggest that data preparation is less important in the age of big data... if anything I would expect quite the opposite. 

Granted, you don't do the classic data warehouse ETL, but every time you try to do something on Big Data you have to figure out how to transform it into something that exposes the information you really want to analyze-- filter out irrelevant records or fields, flatten it or structure it, pre-aggregate it... It's just that now you may need to do it over and over again for every project, what with 'schema on read' and all. (Of course, you only do enough of it to answer the question at hand rather than building the perfect data warehouse... so that's potentially a savings, and an opportunity for a shorter time to first answer vs. waiting on a big bang data warehouse that never finishes. At a minimum, it means spreading out of the costs of prep work across each project that consumes the data.)

We need to:
- simplify the transformation design work to where most business analysts can manage without IT
- help users figure out whether what they have is of adequate quality, i.e. fit-for-purpose, and how to get it there if it isn't already
- democratize the cataloging, curating and searching of the great big bags of bits
- automate the discovery of relevant data and relationships for the specific problem at hand
- enable the identification and re-use of data prep work from other projects where possible, without committing to pre-computing, materializing and constantly refreshing the outputs in case someone wants them

New practices will be developed, new tooling will facilitate all this and really trigger the trip up the Slope of Enlightenment. But in the meantime, many will slog, shoulder-deep, across the trough of disillusionment, a lucky few encouraged by a backhand's brush against some crystallized chunk sunk in the sump, hinting that a toe might soon find purchase on the Slope of Enlightenment... 

But enough of metaphor! Clearly I need sleep. See you all on the Plateau of Productivity...

Hello,
Thank you for your most insightful article! I would like to add some general observations of my own, gathered from work experience and academics in a variety of key functional areas. Please consider this comment as a type of “discussion draft” – here, the goal is to help facilitate discussions about what Big Data is and how we can best utilize it. Yes, “Big Data” is receiving a lot of attention from Senior Managers. Data Scientists, and IT Departments alike. In some ways, Big Data is the confluence of (1) the huge amounts of data being generated every day from new data sources which may not have existed 20 years ago, (2) the dramatic reduction in cost to store this “new” data provided by the Hadoop File System (HDFS), and (3) the development/availability of new technologies and Hadoop Ecosystem tools for storing, managing, and deriving actionable insights from the data. While these technological developments are both impressive and helpful – from my perspective, they may not always be self-sufficient in their own right. For example, from my perspective, some of the most important aspects of understanding and deriving actionable insights from Big Data are (partial list): (a) Metadata. To what degree do we know what the data means and what it does not mean? To what degree can we incorporate qualititative factors in our metadata? To what degree are we able to update and modify (sometimes selectively) metadata as the meaning and/or the structure of the data changes over time? (b) From my perspective, I view the domain knowledge of the Data Scientist as one of the keys to deriving more reliable data insights. For example, when analyzing Federal Reserve Bank economic data, a Data Scientist that understands the complex interrelationships between multi-variable data sets may be better able to provide value in this type of analysis. Here, from my perspective, it is not only the application of inferential statistics, machine learning, heuristics, etc. The Data Scientist, or Subject Matter Expert, may be able to provide perspective and insights, adding value which might otherwise possibly be overlooked. Hopefully, given sufficient interest, we can discuss this and more via future comments.

Very Respectfully,

Steven Gold

MS, Information and Telecommunications Systems, Johns Hopkins University, Baltimore MD

MBA - Marketing, University of Missouri, Kansas City MO

CSO
Strategic IT Security, LLC
srg@strategicitsecurity.com

Couldn't agree more Randy. At Gartner we have compiled hundreds of real-world, high-value use cases of Big Data & advanced analytics, and we're learning of more everyday...particularly as you say Big Data projects are going "mainstream." People often misconstrue Gartner's Hype Cycle. That is, as a technology or technology concept becomes post-peak and starts sliding into the "Trough of Disillusionment" such as Big Data is now, this merely means that the fast-followers and laggards have more difficulty with it than the early adopters--mostly due to cultural, organizational and leadership barriers. Of course this is also when the usual boo-birds and naysayers come out of the woodwork.

Also, note that we have long contended that Big Data is about more than just "volume". My paper from 13 years ago first introduced the three-dimensional challenges and opportunities of increasing data volume, velocity and variety (the "3Vs").  Ref: http://goo.gl/wH3qG. And by 2:1 our clients tell us that data "variety" is the biggest challenge. 

Cheers,
Doug Laney, VP Research, Gartner, @doug_laney