R&D, Meet E&S (Experiment & Scale)

As Internet-driven apps began to overwhelm packaged software business models, Intuit founder Scott Cook painfully, but successfully, reengineered his multi-billion-dollar company’s culture around design experimentation. In fact, he became an experimentation evangelist.

Cook says he “wondered why Google beat Yahoo! at search,” until “a Yahoo! executive told me that Google succeeded by installing the system and culture to decentralize decision-making” and change it to decision-by-experiment.

A few years ago, Google said it ran 3,000 to 5,000 search experiments a year — when you use Google, you are a part of those experiments. Today, that number may be at least 10 times higher.

Cook explicitly links a culture of experimentation to empowerment. A great business experiment has to inspire the same degree of top-management enthusiasm and engagement as a great business plan. Instead of running experiments that validate existing plans and analyses, see them as methods to acquire creative insight and information. That’s when the “a-ha moment” occurs.

When Google engineers have an idea, they don’t need approval to pursue it; they let the experiment make the decision. Google understands that innovators want to take their idea, build it, and see it work.

In today’s real-time, online environment, good ideas matter less; testable hypotheses matter more. Tomorrow’s innovations and strategies will increasingly be the products — and byproducts — of real-time experimentation and testing.

Since the 2001 popping of the Internet bubble, digital networks have successfully evolved into virtual research centers, laboratories, and design studios for entrepreneurs and enterprises to field-test hypotheses that typically required dedicated teams and facilities. As a result, traditional innovation investment paradigms emphasizing Research & Development (R&D) increasingly yield to practices supporting Experiment & Scale (E&S).

The New Economics of Experimentation

Collaborative, open platforms radically reduce cost, risk, and time required to productively run business experiments. Crucially, networked experiments — always ongoing in born-digital enterprises — can swiftly scale into new products, new services, and better user experiences. This effectively converts innovation from long-term, fixed-cost investment to variable or marginal-cost investment. Call it the “exponential economics of networks.”

These new economic models elevate the value of digital experimentation, making it more compelling to mainstream businesses. As experimentation gains acceptance, data-driven top management must re-think innovation opportunity and risk-investment strategies to emulate digital business leaders.

Our work strongly suggests that E&S has become an innovation best practice for digitally sophisticated enterprises; they make experimentation both a core competency and a cultural value.

Amazon, Google, Microsoft, Netflix, Facebook, Intuit and Capital One are just a few high-profile market leaders that publicly attribute innovation prowess to their ongoing commitment to digital experimentation. Firms like these expect innovators to experiment often and freely. Experimental quantity is almost as important as quality. As Amazon founder Jeff Bezos famously observed, “If you double the number of experiments you do per year, you’re going to double your inventiveness.” Networked enterprises can more than double the number of annual experiments with only incremental increases in cost.

Amazon’s recommendation engines, for example, weren’t launched as a grand vision to transform digital shopping; they serendipitously emerged through rapid, agile digital experimentation and scaling — not strategic planning. As former Amazon intrapreneur Greg Linden recalls, real-world results from fast, cheap, and iterative experiments were vital in persuading reluctant top managers to invest in recommendation.

Conquering Management Resistance

This highlights a painful insight: The biggest challenges are not technical or financial, but cultural and organizational. At most firms, management overwhelmingly favors planning, programs, projects, and pilots over the real-world benefits of experimental knowledge and insight. Most don’t realize how exponential economics of experimentation can bolster their innovation investment portfolios.

Executives frequently resist easy opportunities to cost-effectively experiment because they fear challenges to their hard-won professional intuitions and authority. Data-driven digital experiments might undermine pet hypotheses or business perspectives. But preserving, protecting, and defending the status quo may prove even more costly.

To be sure, significant technical distinctions exist, for example, between A/B and multivariate designs. Some organizations define experiments as simply comparing options, i.e., does blue outperform red? Others devise portfolios of fast digital experiments to test rigorously defined strategic business hypotheses. Either way, commitment to experiment early and often — and act on the outcomes and insights — is necessary.

Creativity in the Cloud

The growing dominance of cloud architectures creates global enterprise environments that can further amplify experimental opportunities and effectiveness. Successful clouds are constructed with simple and easy scalability in mind. Similarly, the DevOps movement — linking software development and operations — is explicitly designed to encourage iterative seamlessness between software capabilities and their network deployment. In addition, the rise of big data and effective real-time management of terabytes and petabytes guarantees a wealth of interesting and important correlations worthy of experimental exploration.

Perhaps the most profound and potentially disruptive change in the analytics/experimentation ecosystem is machine learning. Leading-edge data scientists have begun training machine-learning systems to generate interesting hypotheses for experiments.

That is, the systems are being trained to recommend data-driven business hypotheses for marketers, managers, and innovators to experimentally test. In the very near future, the most important experiments — such as improving user experience, or identifying lead users, or suggesting new features/functions — will come from exceptionally well-trained machine-learning systems. Recommendation engines will drive experimental agendas.

Until then, we advise enterprises to consider eight key learnings from our research, as follows:

1. Analyze less, experiment more: Emphasize learning from experiments over predictive analytics for innovation explorations.
2. Testable hypotheses over good ideas: Present testable hypotheses for experimental and/or analytical development.
3. Celebrate creative constraints: Constraints can be sources of enterprise ingenuity — for instance, something that works on a mobile device, an event actuated by a swipe, or a color-change.
4. Make experiments social: Data scientists don’t have to own all experiments. Extend them to enterprise chats or dialogues around innovation, inviting comments and critiques. Socialization — sometimes including suppliers, channels, and customers — will keep constituents aware of ongoing results.
5. Prioritization matters: Eventually, the challenge shifts to prioritizing the portfolio and balancing tensions that typically emerge between marketing/customer-facing managers and technical managers about what to test first.
6. Insights over solutions: Initially, emphasize insights over solutions. Perfectionist engineering is suppressed in favor of quicker, iterative sensibilities.
7. Track the hypothesis trajectory: Don’t just track outcomes; chart the focus of the business hypotheses over time. Do the bulk of hypotheses reflect concerns about customers or suppliers? Channels or partners? What value paths and trajectories do the hypotheses appear to be on? Are they more tactical or strategic?
8. People aren’t lab rats: Ethical considerations and concerns around experiments can’t be ignored. Facebook, for example, created a firestorm of bad publicity and regulatory interest when it tweaked user newsfeeds — without their knowledge or consent — to explore hypotheses around networked emotional contagion. Assuring that customers are respectfully treated is a paramount concern.

You can read the full research report here