Big Data Lessons at MIT

Alex (Sandy) Pentland, director of MIT’s Human Dynamics Laboratory and MIT Media Lab Entrepreneurship Program

Companies are now amassing so much data with such variety, so quickly, that they are able to peek beneath their organizational skin, much as biologists in the 17th century used microscopes to acquaint themselves with the (then) unknown multicellular denizens of pond water. What can companies see with all of this new data?

By describing our world at a much finer level of detail, Big Data is improving our ability to ask new questions, to solve old problems and to innovate. It is also enabling higher levels of corporate productivity. Perhaps, most importantly, it is forcing us to make trade-offs about the kind of world we want to live in.

Erik Brynjolfsson, director of MIT’s Center for Digital Business

These are some of the messages that MIT professors Erik Brynjolfsson and Alex (Sandy) Pentland drove home in a thought-provoking two-day seminar called Big Data: Making Complex Things Simpler, held near the MIT campus last month.

Brynjolfsson is director of MIT’s Center for Digital Business and co-author of Race Against the Machine: How the Digital Revolution is Accelerating Innovation, Driving Productivity, and Irreversibly Transforming Employment and the Economy. Pentland directs MIT’s Human Dynamics Laboratory and the MIT Media Lab Entrepreneurship Program, and also advises the World Economic Forum, Nissan Motor Corporation, and a variety of start-up companies.

A few of the larger implications discussed at the two-day event are continuing to resonate with me weeks later, including:

• Faster insights with cheap experiments. A significant challenge for online retailers is abandoned shopping carts. People stuff their virtual carts with items they never actually purchase. How to minimize abandoned shopping carts and increase sales?

At Amazon, data analysts are running daily experiments: changing prices for a given item or presenting items in different ways. With cheap flows of data, the cost of running these experiments is low, so Amazon can run hundreds of inexpensive experiments each day. Most fail to show improvement.

But in this case, failure is a sign, paradoxically, of success. As experiments get cheaper, you can run more and more experiments – so the ones that work will get fewer and fewer as a percentage of total experiments run. “That’s not a bug,” said Brynjolfsson. “That just shows it’s worth testing even long-shot hypotheses.”

• Big Data/Big Brother? Seminar participants had a lot of energy and concern around the privacy implications of Big Data. They discussed a number of intriguing questions: What exactly are companies doing with mobile data, for example, which can show the current and historical information about an individual’s location? How much control should an individual have over data that connects to his or her behavior? If it’s your data, can you sell it? Is personal data a kind of new asset class? What kind of markets might there be for this information? How would they be regulated?

These are important questions for all of us to consider in the emerging era of Big Data. In a future post, I’ll blog about other management implications discussed in Brynjolfsson’s and Pentland’s seminar, which will be taught again in October 2012.