It may be everywhere now, but big data matters more than ever.
I was having lunch recently with the editor of a leading business publication, when I was asked, “Don’t you see a backlash against 'big data'? Aren’t people growing tired of hearing about it?”
This seemed a fair question, given the sheer volume of articles and media coverage on the topic of big data during the past three years. But my response was a resounding, “No!”
During the summer of 2013, NewVantage Partners conducted a survey of Fortune 1000 C-suite executives from companies including American Express, CVS Caremark, JP Morgan, Johnson & Johnson, Kaiser Permanente, MetLife, Travelers and Wells Fargo, among others.
Our survey found that 91% of executives indicated that they have a big data initiative planned or in progress, with 60% reporting having an initiative completed.
Let’s consider three reasons why big data is becoming part of the mainstream now.
1. Big data is about all data, not just social media, unstructured or massive data.
I’m sometimes told by senior corporate executives that they don’t have a big data need because they are not focusing on social media data, unstructured data or massive data sets. This is a common misconception about big data. While much of the talk about big data focuses on the benefits and opportunities that result from new sources of data — including social media, sensor and visual data — most of the action among mainstream corporations is focused on integrating information from traditional legacy environments, like COBOL and mainframe data sources.
Integration of legacy data remains one of the main challenges for most corporations born before the digital era. When asked what kinds of data corporations planned to integrate using big data capabilities, most respondents to our survey responded that their focus was on customer transaction and financial data. For corporations, the ultimate ability to link behavioral, transaction and customer interaction data provides insight into the relationships between what customers say and what they do.
Similarly, big data is not exclusively about massive data sets. It is often assumed that big data is exclusively about capturing very large volumes of data. But big data is also about integrating more sources (“variety”) of data, which needn’t be massive. As Babson College's Tom Davenport states in a 2012 blog article, “Even small data can improve your organization’s judgment.”
Big data is about all data — big and small, structured and unstructured, new and legacy. And, for mainstream firms, executive respondents to our survey indicated that integrating and analyzing data from existing data sources was their greatest priority. In short, any organization that has “a lot of data” of any size, shape, form or variety has a potential big data opportunity.
2. Big data will change the time and cost equation for all data applications.
For many mainstream corporations, big data is no longer an experiment. Some context and history may help explain why.
Big data refers to a set of data management technologies, such as Hadoop, that were first employed by social media companies including Google, Facebook and Yahoo to enable the processing of massive volumes of information in a timely fashion.
Speed is realized through the ability to shorten the cycle from data access to analytical results. This is the result of the ability to load “all of the data” for quick and easy analysis, foregoing time-consuming processes or data engineering and up-front hypothesis formulation. Big data makes it possible to get more done for less by lowering the cost of managing data in two principle ways:
- Reducing the need for costly upfront data preparation and data engineering, which typically constitute 80% of the time and cost of data management, and
- Using lower-cost big data platforms like Hadoop, built with open source software, which are typically a fraction of the expense of traditional database platforms. (These numbers are based on our work with top financial firms and their own self-reports about cost advantages.)
Big data is becoming an integral part of the mainstream for large corporations as they come to understand that they can, for the first time, make data of any kind available to business analysts in a timely and cost-effective fashion. For these corporations, big data means faster time-to-market and quicker response to customer needs and interests, and the opportunity to accelerate time-to-value through greater speed and agility.
As big data approaches gain-growing acceptance, many mission-critical operational and regulatory reporting applications will be candidates for migration. Higher-cost production processes will be migrated to lower-cost platforms. This will be an evolutionary process, playing out over the course of the coming decade, as firms establish an evolving data-management ecosystem that combines traditional data-management approaches and new big data approaches.
For companies on the forefront of using big data, this migration of mission-critical processes is already well underway. Because big data represents an inexpensive way to get fast results, the need and the demand will only increase.
3. Big data is a term that captures the zeitgeist.
Love it or hate it, big data is a term that has caught on. But forget the semantics; focus on the benefits. In spite of lingering misconceptions, big data will compel interest and drive business value for many years to come.