Analytical Value From Data That Cries Wolf

Imperfect data can still be put to good uses.

Data can lie. Sometimes it tells you things that just aren’t true. A positive cancer screening, for example, induces panic until later investigation reveals the result was a false alarm — a false positive.

Like medical tests, data used as input for corporate analytics can contain false positives. And, like medical tests, these false positives can lead to incorrect analysis and unnecessary action. For organizations relying on analytics, accepting fiction as fact may be dangerous.

Yet, not all false positive are poisonous; despite the risk, many companies find ways to get value from data filled with false positives. Here are some guidelines for creating value from false positives.

Understand the trade offs between false positives and false negatives.

Ideally, data contain neither false positives nor false negatives. In reality, data sources balance the two; decreasing one usually means increasing the other. For example, consider news. A newspaper can expend considerable effort to validate information from multiple independent sources before publication. To avoid unnecessary publication cost and reputation damage, the design of the process emphasizes publishing articles that are accurate. However, this accuracy comes at the expense of false negatives. By working so hard to avoid publishing mistakes, a newspaper may miss or delay real stories. As a data source, newspapers avoid false positives, but in doing so they sometimes are left with false negatives.

In contrast, Twitter users rarely miss anything — for example, the raid on Bin Laden was live tweeted long before mainstream news covered the story. Yet inaccurate (and often completely unfounded) rumors run rampant as well — many celebrities must have feline ancestry to bounce back from their many Twitter-based death reports. With low production cost and minimal reputational concerns, this data source offers insights and speed unavailable in the opposite approach. But the tradeoff is an increase in false positives.

Identify the sensitivity and specificity of each data source.

Organizations need to assess both the rate of true positives (sensitivity) and true negatives (specificity) for each data source to understand inherent tradeoffs. Does this data source lean towards false negatives or false positives? For example, data from hotel reservations guaranteed by advance, nonrefundable payment will likely have few false positives; this data source will have few reservations that go unused, but it may underestimate actual demand. Alternatively, systems that offer no-cost reservations are more likely to reflect hotel stays that do not actually happen. Models to estimate demand based on each data source would need to be different depending on the data source. Demand estimates based only on website searches for availability would differ even more. Potential tradeoffs cannot be incorporated into analysis until identified.

Propagate underlying uncertainty throughout the analysis.

Once source tradeoffs are identified, analytics results should reflect this underlying uncertainty. Disciplines such as chemistry and physics have rich histories of error analysis, with guidelines for aggregating error and propagating significant digits. However, in organizations, estimates may be presented as simple numbers that, in aggregate, obscure inherent bias that results from the data sources. Uncertainty in a result is neither indecisive nor indicative of a problem; the only real problem is thinking that there is certainty when there isn’t. As analytical thinking increasingly spreads throughout organizations, so too should comfort with analyses that incorporate uncertainty.

Seek variety for a portfolio approach.

If all the data in your analysis is from data sources that traditionally focus on reducing false positives, a data source that allows for false negatives adds insight. Choosing one or the other is a false dichotomy. In the hotel reservation example, the estimates of demand that combine prepaid reservations, no-commitment booking, and online availability search will outperform estimates based on only one source because each of the three data sources offers unique perspectives that, when combined, yield better insight to the true demand. Restaurants are embracing this approach by combining manual reservations from loyal customers, data from Open Table, and managerial insight.

In many contexts, predictive analytics is embracing data sources with a variety of perspectives. Uber blends multiple uncertain inputs to predict rider destinations and improve service. Looking at desserts on the menu doesn’t always lead to an ice cream order, but analytics at McDonalds incorporates the probability that it will. Airplane arrivals may not always translate to rental car demand, but Enterprise uses this signal to adjust staff and minimize wait. Even data rich in false positives can inform.

Know where the uncertainty lies when making decisions.

Previously, when information was more difficult and expensive to produce, organizations had experience to guide them in how to incorporate it into managerial decisions. Now, organizations must adapt to an environment of cheap and fast dissemination of dubious information. Yes, ideally organizations should work to minimize both false positives and false negatives. However, rather than ignoring data sources with false positives, analytics can use them to create value. The key is understanding the data; simple examples such as the Deming Funnel experiment illustrate that managing without understanding data is worse than doing nothing.

3 Comments On: Analytical Value From Data That Cries Wolf

  • Keith Drummond | September 30, 2014

    Good article, and I certainly agree with the idea of combining several data sources to try to paint the true picture.
    Establishing a link between the different data sources is perhaps another consideration , that plays into how we weight one data source over another.

  • Richard Ordowich | October 1, 2014

    Determining which results are false positives and false negatives itself is a challenge. You don’t know beforehand. Data analytics are useful but testing for false positives and negatives requires a process that does not use the data as a verification source.

    The results of analytics should be scrutinized using what humans are good at, the “smell test” or intuition. Where did the data come from? Do we trust the data and those who created the models? What are their motivations? In research studies most senior management decisions are made not based on existing data but on networking with colleagues and acquiring new data sources.

    McDonald’s was suffering from a decease in same store sales and used their data to try and determine why. Unfortunately there was nothing in the data except for confirming a decrease in sales. What they determined is that customers were dissatisfied with the customer experience. Slow and unpleasant interactions resulted in customers not coming back. There was no data available to determine the cause but a visit by management would have easily exposed why sales were down.

    Too many people assume data is “real” that data represents “facts”. But to quote a philosopher; “all data lies”. Extracting the story, fiction or non- fiction from the data is the challenge and using the same data to determine that isn’t very helpful. Many analytics are in support the story someone wants to tell. As a result data selection is biased.

  • HR Harvard | October 9, 2014

    Data is what links our abstract world into the granularity of things we understand and relate to. Before data becomes useful information, It must be measured, stored, manipulated and displayed. Each one of these phases is a combination of reliable and unreliable methods. Human subjectivity is needed in many cases, when technology alone can’t simulate circumstantial and/or environmental factors.

    I like how this article talks about having data sources to be as versatile as possible to cover the most part of what influences data. Also, I agree completely on the benefit of knowing the rate of false positives/negatives with respect to data sources, which could lead to better decision making.

Add a comment