Just before the New Year, New York Times writer Steve Lohr wrote a blog post, Sure, Big Data Is Great. But So Is Intuition, which addresses a question we here at MIT Sloan Management Review have been researching for the past year: With the growing potential of data and analytics, where is the shifting line between analytics and intuition?
In other words, is there a “correct” balance between analytics and intuition in making good business decisions?
Lohr bases his post on comments made by a number of speakers who presented at a recent MIT Center for Digital Business conference, Big Data: The Management Revolution, including Claudia Perlich, chief scientist at Media6Degrees and Rachel Schutt, a senior statistician at Google Research. His central premise:
Personally, my…concern is that the algorithms that are shaping my digital world are too simple-minded, rather than too smart.
It’s encouraging that thoughtful data scientists like Ms. Perlich and Ms. Schutt recognize the limits and shortcomings of the Big Data technology that they are building. Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?
As I read Lohr’s article, I couldn’t help but think of our own findings around the theme of analytics versus intuition.
In conjunction with our corporate partner SAS, we have conducted research into how organizations are using data and analytics to both compete and innovate. As part of that research, we asked 2,500 survey respondents to rate the extent to which they rely on intuition versus analytics to make decisions in a number of areas, from enhancing customer experience to allocating annual budgets.
We found that while organizations have access to more useful data than they did a year ago — another finding from our survey — the extent to which managers are using that data to make business decisions varies greatly (see “Addressing Analytics Equilibrium”).