Embrace Your Ignorance

Today’s savviest executives acknowledge that most of the time, they can’t accurately predict what their customers will want, like or need. That’s where experiments come in.

Reading Time: 4 min 


Permissions and PDF

Microsoft’s Ronny Kohavi, a pioneer in online experimentation since his days as director of data mining and personalization at Amazon.com, likes to challenge his Internet-savvy audiences with a little experiment. Kohavi, a Microsoft distinguished engineer and general manager for analysis and experimentation, shows screenshots of simple but real A/B experiments that Microsoft ran to test different Web design options. Some visitors saw the A version, some the B version — and Microsoft tracked which version yielded better results. Kohavi then challenges his audience to predict the real-world outcomes of the test: Did Option A win? Did Option B win? Or were the results a statistical dead heat?

Almost without exception, his audiences fragment; not even a rough consensus emerges. “Stop debating,” Kohavi advises. “It’s easier to get the data … Data trumps intuition.”

Data from experiments such as Kohavi’s should ideally encourage executive curiosity and humility. Unfortunately, that rarely happens. Too many executives worldwide would rather use their expertise to “think” or “intuit” what’s best for their customers than run even the simplest experiment. They take pride in their “gut.” That’s a huge managerial mistake.

Organizations may be confident they know their customers, but they’re very likely to be overconfident. Most executives aren’t nearly as smart, perceptive or customer-centric as they believe. Leading researchers, such as Daniel Kahneman, who received the 2002 Nobel Prize in Economic Sciences, and psychologist Philip E. Tetlock, have persuasively demonstrated that “experts” are particularly prone to predictive overconfidence.

For example, in a study where experts were asked to make predictions related to world events, Tetlock reported in a 1999 article that “across all seven scenarios, experts were only slightly more accurate than one would expect from chance.” What’s more, Tetlock found that the “experts who assigned confidence estimates of 80 percent or higher were correct only 45 percent of the time.” The overconfidence — arrogance? — of presumed expertise is counterproductive. Pride goeth before a fall. Data trumps intuition. Serious innovators take data seriously.


Reprint #:


More Like This

Add a comment

You must to post a comment.

First time here? Sign up for a free account: Comment on articles and get access to many more articles.

Comments (3)
For a number of year, large supermarkets have been running A/B experiments, trialling stock locations at various instore positions to maximise sales. Huge data gathering initiatives drive this, monitoring sales against data collated from loyalty programmes, seasonal influences and even daily cycles.  Data directs these activities, encouraging best product placement to encourage sales. Why not for every business?
Seth Bauer
Most of those A/B website-type experiments, though, are based on very limited amounts of data.  How reliable are they?
Wonderful stuff, thank you Michael.  Somehow I missed your latest book...just ordered it!