Variety, Not Volume, Is Driving Big Data Initiatives

When many executives think of Big Data, they think of large volumes of data. A common notion is that bigger is often better when it comes to data and analytics, but this is not always the case. In their 2012 article, Big Data: The Management Revolution, MIT Professor Erik Brynjolfsson and principal research scientist Andrew McAfee spoke of the “three V’s” of Big Data — volume, velocity, and variety — noting that “2.5 exabytes of data are created every day, and that number is doubling every 40 months or so. A petabyte is one quadrillion bytes, or the equivalent of about 20 million filing cabinets’ worth of text. An exabyte is 1,000 times that amount, or 1 billion gigabytes.” This focus on the rate of data proliferation has sometimes obscured an appreciation of data and analytics value. The result is a myth about Big Data — that Big Data is synonymous with large volumes of data.

In 2012, when Brynjolfsson and McAfee published their article, Big Data was a new phenomenon. While a handful of Silicon Valley innovators like Google, Facebook, and Amazon were employing Big Data with success, Big Data was largely uncharted territory for mainstream Fortune 1,000 firms. The past several years have been period of exploration, experimentation, and trial and error in Big Data among Fortune 1,000 companies, and the result has been a different story. For these firms, it is not the ability to process and manage large data volumes that is driving successful Big Data outcomes. Rather, it is the ability to integrate more sources of data than ever before — new data, old data, big data, small data, structured data, unstructured data, social media data, behavioral data, and legacy data.

This is known as the “variety challenge,” and has emerged as the top data priority for mainstream companies, according to the fourth annual Big Data Executive Survey, conducted by NewVantage Partners and released last month. In the world of the Fortune 1,000, we are seeing that variety trumps volume and velocity when it comes to Big Data success.

Tapping Into the “Long Tail” of Big Data

When asked about drivers of Big Data success, 69% of corporate executives named greater data variety as the most important factor, followed by volume (25%), with velocity (6%) trailing.