The Perils of Applying AI Prediction to Complex Decisions

Leaders are increasingly aware of AI’s potential to reshape the business landscape. But they may still lack an understanding of how to apply it within their organizations, or they might expect that it can solve more complex problems than technology currently allows. A study in 2012 conducted by eBay Research Labs economists looking at eBay’s digital advertising strategy showed the perils of blurring correlation and cause. The study revealed that certain digital ads, which had appeared hugely effective for eBay, were not actually effective returns on investment. The ads hadn’t brought the e-commerce giant any new customers because people who saw them were already going to go to the site.

These are the sorts of quandaries machine learning research is not yet able to disentangle, according to Susan Athey, an economics of technology professor at the Stanford Graduate School of Business, who cites the eBay research in a paper published in Science. Athey says that to reap the benefits of the new technology, business leaders need to figure out which are the right problems to apply it to.

MIT Sloan Management Review correspondent Frieda Klotz spoke with Athey about the perils of applying AI in the wrong contexts. What follows is an edited and condensed version of their conversation.

MIT Sloan Management Review: What are some of the common challenges people have with understanding the benefits of machine learning?

Athey: There’s a misconception that it’s always going to be better to let an algorithm determine a solution, but that won’t always be the case. AI isn’t a good fit for every sort of problem. When it comes to automation, a real challenge lies in understanding what are the easy tasks and what are the hard ones. Take playing a board game, in which AI technologies have famously been successful. In some ways this is a difficult task because the state space is very large — that’s the term we use to describe the possible sequence of moves available to the machine in the game — but it’s also easy in the sense that the task itself is well-defined. The mapping from the play to the outcome is clear, and at the end, there’s no doubt about who won and who lost.

Some real-world scenarios are like that, where you can simply repeat something over and over, and test it without bearing any costs. But often if you don’t fully understand the mapping between actions and payoffs, you have to interact with the real world to gather data, and then your business will potentially bear real costs. Or the timing makes it impossible. While a computer can play a board game very quickly, in the real world, if you’re trying to understand the effect of a decision whose outcome occurs six months later, it will take years to test the nature of the results. So a lot of real-world problems don’t share the same characteristics as those portrayed in media discussions about AI and machine learning.

Have organizations gotten carried away with the promise of machine learning?

Athey: I try to help managers understand what makes a problem easy or hard from the perspective of AI, given the sort of AI that we have today. I want them to recognize where the new technology will be awesome and where they are really going to need different tools.

The truth is that when firms start to use machine learning they usually do so in a pretty simple way involving routine problems. The more complex AI is used in a much smaller set of categories — important categories like self-driving cars — but even industrial robots are generally doing routine, repetitive tasks in a very controlled environment. Using machine learning for a general-purpose task in a very unstructured environment is much less common.

It seems as though it’s vital for companies to experiment and test their data if they want to avoid pitfalls.

Athey: Absolutely. These issues come up especially in online advertising. If a company puts an ad in front of people and it reminds them to do something they were already planning to do, it just reminds them to do it right at that moment. So the ad will look very effective when it’s not effective at all. For the set of people who type “eBay” into a search engine, what’s the chance they were going to go to eBay without the ad? Very, very high.

The people most likely to engage with your product might be those who are not influenced by marketing activities or sales activities. A lot of newer use cases of new machine learning technologies have that lens.

What are some examples you have come across?

Athey: Some machine learning services claim to identify which customers a call center should target. What a business in this scenario really wants to know is the ROI — to compare what happens if they make the call or don’t.

But these systems are predicting which customers, when called, will purchase. So you’re basically predicting purchase, conditional on call. This is a much simpler problem and one that is easier to solve. Ultimately, a lot of those customers might have purchased anyway, so you’re really predicting which people like your product the most, which is not the same as predicting the people for whom a call will make a difference in whether they make a purchase. And people for whom the call is an effective push might also have lower purchase probabilities, so the picture becomes increasingly complex.

On the other hand, it’s incredibly useful if you can predict the people who will never buy your product no matter what you do because then you can get rid of those calls. It’s among the people who might buy that prioritizing is a harder problem and one that might require an experiment.

So it’s really about asking the right questions.

Athey: Thinking critically about how you gather data and approach the data you have is very important. Here’s another example: It’s common in data sets to see a positive correlation between price and quantity. Typically for hotels or anything that sells out, the period when they are likely to sell out is also the time at which sellers raise prices.

If you look at that data, you might think it was saying that if you raised prices it would help you sell out your hotel, when, of course, the causality runs in the opposite direction — it’s because it’s a busy period that the prices are high. [Businesspeople] have understood for decades that it’s very hard to use historical data to make pricing decisions. Traditional companies will run experiments or do surveys when coming up with solutions.

But then the question is: What do you do? In this case, the data alone will not guide hotel owners to set prices in a way that will increase sales.

Businesses need the right kind of data, and they need to have experimental variation of some sort. More bad data doesn’t solve the problem. People often think that bad means messy or mismeasured, but it’s not about mismeasurement, it’s about finding data that can help determine what would have happened if you had set a different price for the same sort of product.

How do you think business leaders should approach the new generation of machine learning researchers you mentioned, who are coming into the workplace?

Athey: Leaders need to understand when it will benefit them to use an off-the-shelf machine learning algorithm and when to apply it. There are many tasks where this strategy is useful. In situations where it’s very clear how the algorithm is performing, or where you have very precise measures, you can hire somebody right out of school, and they’re going to create enormous value.

In other cases, it can be trickier. For instance, when there are these long-term measures or when there are issues around correlation and causality. … That’s when you want to think about machine learning more as a capability and not as a whole system. To determine the right solution, you will need people with domain expertise who really understand the problem in the context of the company. These are the people in your organization who understand the challenges and have a more conceptual approach, and they can be paired up with younger machine learning practitioners who are enthusiastic to bring in new machine learning solutions.