Embrace Your Ignorance

Microsoft’s Ronny Kohavi, a pioneer in online experimentation since his days as director of data mining and personalization at Amazon.com, likes to challenge his Internet-savvy audiences with a little experiment. Kohavi, a Microsoft distinguished engineer and general manager for analysis and experimentation, shows screenshots of simple but real A/B experiments that Microsoft ran to test different Web design options. Some visitors saw the A version, some the B version — and Microsoft tracked which version yielded better results. Kohavi then challenges his audience to predict the real-world outcomes of the test: Did Option A win? Did Option B win? Or were the results a statistical dead heat?

Almost without exception, his audiences fragment; not even a rough consensus emerges. “Stop debating,” Kohavi advises. “It’s easier to get the data … Data trumps intuition.”

Data from experiments such as Kohavi’s should ideally encourage executive curiosity and humility. Unfortunately, that rarely happens. Too many executives worldwide would rather use their expertise to “think” or “intuit” what’s best for their customers than run even the simplest experiment. They take pride in their “gut.” That’s a huge managerial mistake.

Organizations may be confident they know their customers, but they’re very likely to be overconfident. Most executives aren’t nearly as smart, perceptive or customer-centric as they believe. Leading researchers, such as Daniel Kahneman, who received the 2002 Nobel Prize in Economic Sciences, and psychologist Philip E. Tetlock, have persuasively demonstrated that “experts” are particularly prone to predictive overconfidence.

For example, in a study where experts were asked to make predictions related to world events, Tetlock reported in a 1999 article that “across all seven scenarios, experts were only slightly more accurate than one would expect from chance.” What’s more, Tetlock found that the “experts who assigned confidence estimates of 80 percent or higher were correct only 45 percent of the time.” The overconfidence — arrogance? — of presumed expertise is counterproductive. Pride goeth before a fall. Data trumps intuition. Serious innovators take data seriously.