The Problem With Big Data

As technology improves, larger companies continue to gain disproportionate shares of the processing power and financial value big data offers.

Reading Time: 4 min 


Because of big data — a term that has come to refer to the immense amount of digital material we generate, store, and manipulate with increasing ability — managers can measure more about their companies and then use that information to drive performance. Need to heighten the productivity of your workforce? Big data can help. Want to analyze customers’ preferences and purchase patterns? Big data can do that, too. Looking for ways to cut costs and increase profitability? Big data: At your service.

But not all companies are flourishing in this new era. Small companies are struggling. Over the last three decades, the annual rate of new startups has fallen from 13% to less than 8%. During that time, the percentage of employment at companies with fewer than 100 workers has decreased by 5%. Meanwhile, big companies are thriving. The share of revenue of the top 5% of businesses has increased by 10% since the 1980s. Large companies also employ a greater share of the U.S. labor force: from one-quarter in the 1980s to about one-third today. What accounts for this discrepancy?

My colleagues — Juliane Begenau of Stanford University’s Graduate School of Business and Laura Veldkamp of Columbia University Business School — and I had a hunch that one of the likely culprits was big data. Our reasoning: Investors rely on big data to help them make smarter investment decisions. Because big companies produce more data relative to smaller companies, investors have more information to go on. This wealth of data, in turn, accelerates advances in processing speeds and computing and helps investors view these large companies as a less risky bet. As a result, big companies get more than their fair share of financing at better terms and are therefore better able to prosper. Smaller, younger companies, by contrast, receive less financing, which hampers their ability to grow.

To test our hypothesis, we developed a noisy rational expectations model to demonstrate how big data influences the evolution of company distribution. Our model calculates the different availability of data in small and large companies and then looks at how that data interacts with the growth of processing power.

To understand just how rapidly it has changed, consider these striking facts: Data processing power has grown about 20% per year since 1980, where our research begins.


More Like This

Add a comment

You must to post a comment.

First time here? Sign up for a free account: Comment on articles and get access to many more articles.

Comments (2)
Prakash Shrivastava
Could there be Limit to Growth in Actionable Insights ?

The positive reinforcement cycle of data growth would continue. But  the rate of how much valuable actionable insights can be gained from this enormous data might become limited. There would be some counterbalancing that will come into play.
Luke Williams
Google is an example of a company that is becoming, I think, somewhat overwhelmed by big data. With now literally billions of web pages (and I'm using "literally" correctly here), constantly crawling those pages, indexing their content, and continually refining an already complex series of algorithms to come up with a ranking for any conceivable phrase a searcher enters in Google search is becoming a daunting task. I think it shows in search results, which I think have declined in quality the last couple of years.

I know a guy who does SEO at, and he agrees with this. It's a sort of blessing and a curse from his perspective. There are more panicky site owners looking to get their rankings back up (more business for him), but trying to figure out how Google's search algorithms work is harder (so more effort for him). I don't even know if Google fully knows exactly how their algorithms work any more!