Much has been written about the high failure rate of data science projects. Data science teams often have difficulty moving their insights and algorithms into business processes. At the same time, business teams often can’t articulate the problems they need solved. And they ignore their cultural resistance to data science.
There is a designed-in structural tension between business and data science teams that needs to be recognized and addressed. Structural problems demand structural solutions, and we see a way forward through a data science bridge: an organizational structure and leadership commitment to develop better communication, processes, and trust among all stakeholders.
Get Updates on Leading With AI and Data
Get monthly insights on how artificial intelligence impacts your organization and what it means for your company and customers.
Please enter a valid email address
Thank you for signing up
To understand the challenges, we use the metaphor of a factory (standing in for any sort of business operation) and a data science lab (such as R&D). The “factory” might literally be a factory, but it could also be a business operation that produces decisions on mortgage applications, reads MRIs, or drills oil wells. Factory goals may involve meeting production plans and keeping unit costs low. The watchword for these operations is stability: no major disruptions and few surprises. Factory managers work very hard to establish and maintain stability, so quite naturally they resist anything that threatens it. One important exception is that good factory managers support incremental improvement when it’s quickly followed by stability at a new level, provided that it is driven within the factory. We call these efforts small data projects.
The data science lab, on the other hand, is specifically designed to disrupt the factory. Its job is to find bigger improvements than can be made with small data, change the way decisions are made, and come up with new products that obviate old ones — with a focus on large, complex, unstructured, and data-rich problems. The lab represents the very antithesis of stability — exactly the sorts of things that factory managers hate!
A certain amount of friction between any two departments is to be expected. Factory managers do not appreciate interference from finance when it changes accounting practices, or from HR when it mandates improvements to its employee review system. But these are mere annoyances; they don’t change the ways the factory fundamentally does its core work.
Contrast these annoyances with the full-frontal attack and disruption signaled by the lab.