The Problem With Big Data

Because of big data — a term that has come to refer to the immense amount of digital material we generate, store, and manipulate with increasing ability — managers can measure more about their companies and then use that information to drive performance. Need to heighten the productivity of your workforce? Big data can help. Want to analyze customers’ preferences and purchase patterns? Big data can do that, too. Looking for ways to cut costs and increase profitability? Big data: At your service.

But not all companies are flourishing in this new era. Small companies are struggling. Over the last three decades, the annual rate of new startups has fallen from 13% to less than 8%. During that time, the percentage of employment at companies with fewer than 100 workers has decreased by 5%. Meanwhile, big companies are thriving. The share of revenue of the top 5% of businesses has increased by 10% since the 1980s. Large companies also employ a greater share of the U.S. labor force: from one-quarter in the 1980s to about one-third today. What accounts for this discrepancy?

My colleagues — Juliane Begenau of Stanford University’s Graduate School of Business and Laura Veldkamp of Columbia University Business School — and I had a hunch that one of the likely culprits was big data. Our reasoning: Investors rely on big data to help them make smarter investment decisions. Because big companies produce more data relative to smaller companies, investors have more information to go on. This wealth of data, in turn, accelerates advances in processing speeds and computing and helps investors view these large companies as a less risky bet. As a result, big companies get more than their fair share of financing at better terms and are therefore better able to prosper. Smaller, younger companies, by contrast, receive less financing, which hampers their ability to grow.

To test our hypothesis, we developed a noisy rational expectations model to demonstrate how big data influences the evolution of company distribution. Our model calculates the different availability of data in small and large companies and then looks at how that data interacts with the growth of processing power.

To understand just how rapidly it has changed, consider these striking facts: Data processing power has grown about 20% per year since 1980, where our research begins.