Breakthroughs and the ‘Long Tail’ of Innovation

Many managers have little understanding of the process of invention. Nor do they possess much insight about the most likely sources of technological and scientific breakthroughs. Specifically, are blockbuster innovations more likely to come from a lone inventor or from a collaborative team? If it’s the latter, does greater diversity on the team help or hurt the group’s chances? And does a deeper understanding of science lead to more breakthroughs, or is such knowledge more likely to result in only incremental progress?

To answer such questions, managers first need to understand that invention is essentially a process of recombinant search. That is, I adopt the classic definition of invention as a new combination of components, ideas or processes. At its simplest level, this definition provides an accessible picture of the inventor as a tinkerer, trying different combinations of materials, gadgets and configurations, and every invention can be thought of as an assemblage of its constituent parts, including the steamship (sailing ship and steam engine), the automobile (bicycle, carriage and internal combustion engine) and Apple Inc.’s iPod (cheap memory, digital music and lightweight battery).

The prevailing view is that breakthroughs are impossible to predict, but that’s only partly true. Much of the misconception arises because people tend to focus on just the breakthroughs while ignoring the iterative process of invention and the resulting total distribution of outcomes. When all inventions (that is, all new combinations) are considered, they demonstrate a highly skewed distribution. (See “Histogram of Creativity.”) Almost all inventions are useless; a few are of moderate value; and only a very, very few are breakthroughs. Those breakthroughs constitute the “long tail” of innovation.

Every well-sampled distribution of inventive value, creativity or success I have ever observed has demonstrated that highly skewed profile. This holds true whether the measure is patent citations,1scientific citations,2 financial returns3 or the number of times a novel combination is used by future inventors.4 Interestingly, it also characterizes other creative fields, such as the number of performances of a musical composition, the number of times a book is reprinted or the value accorded to a collector’s comic book. 5

The skewed distribution, however, is rarely acknowledged, let alone investigated and systematically managed. That probably occurs for a number of reasons, including the difficulty of collecting complete data (failures are often forgotten), the statistical difficulties caused by long-tail distributions and the convention of discarding outliers rather than learning from them. Yet if managers wish to understand how those outliers (that is, breakthroughs in creativity) arise, they cannot ignore the process that generates the entire distribution. In particular, they need to keep in mind the following three measures:

1. Shots on goal: The first step in inventing a breakthrough is generating numerous draws because single attempts rarely become one-hit smashes. If invention is a process of recombinant search, then generating draws requires putting together a lot of previously untried combinations. Each new one is an invention and provides a shot at a breakthrough.

2. Average score: Once a company is generating a sufficient number of shots on goal, the next step is to maximize the average score of those shots. (Here, the use of a sports metaphor is misleading. Unlike basketball, soccer and other sports, innovation is an endeavor in which the shot scores vary greatly in their value. The value of a breakthrough invention can easily be several orders of magnitude greater than the value of a mediocre one.) A crucial thing to remember is that organizations rarely invent a breakthrough if their average shot is worth little.

3. Maximum scores: If a company truly desires a breakthrough, it needs to do more than simply increase the number and average score of its shots. It also needs to substantially expand the variability of those scores. In other words, it needs to take wild shots at a rich target (or, preferably, a set of rich targets) because the wider range will be more likely to contain scores of maximum values.

With those three measures in mind, managers can gain a deeper understanding of how breakthroughs truly occur. Moreover, they will be able to obtain a better appreciation of the different factors that affect the process of innovation, enabling them to manage that process to increase their odds of success.

Lone Inventors As the Source of Breakthroughs: Myth or Reality?

One very persistent belief is that lone inventors are the source of breakthroughs. That notion was popularized around the turn of the 20th century when lone mavericks were credited with a host of breakthroughs in communications (Alexander Graham Bell), utilities and entertainment (Thomas Alva Edison) and transportation (Henry Ford), among other industries. But are lone inventors truly the source of such quantum leaps in technology, or is that belief more myth than reality?

My fieldwork consistently indicates that innovators working by themselves can be the source of more failures as well as more breakthroughs. Their output at both extremes of the distribution suggests that the impact of lone inventors’ work is highly variable. To test that hypothesis, I looked at a 10% sample of all patented U.S. inventors since 1975. The models indicate that lone inventors generate fewer novel combinations and that the combinations they create are less likely to be used, on average, by future inventors. Those results call into question whether lone inventors are truly the creative geniuses that they are reputed to be. But the future use of their work is also much more variable, such that they are more likely to be the source of a highly skewed outlier, thus bolstering the argument that they are indeed more likely to be the sources of radical innovations.6

One possible explanation is that loners generate fewer novel combinations because most inventors receive their information from social sources.7 As a result, collaborative inventors are exposed to more recombinant opportunities and hence are able to contrive a greater number of new combinations. A reason that collaborators have a higher average score than loners might be because collaborators help one another identify the most promising new combinations for further development: They make the selection stage of invention more rigorous, thus improving the average quality. Inventions by collaborators are also more likely to be adopted by others because there are a greater number of diffusion paths for that knowledge to travel. (For noncollaborative innovations, the lone inventor is usually the sole source of the necessary expertise.)

A less rigorous selection process for loners, however, also means that they will have a greater variance of output. Because they are less constrained by convention and skeptical groupthink, lone inventors are more likely to invent (and not immediately dismiss) the radical breakthrough. Thus, lone inventors are less creative on average and yet are also more likely to come up with a breakthrough. In other words, lone inventors make fewer shots on goal, have lower scores on average and tend to score both very low and extremely high. They are on average both less successful — and more likely to be the source of breakthroughs.

Since 1975, lone inventors have been responsible for more than 20% of all patents held by U.S. corporations.8 As such, they are an important resource in the corporate lab despite their reputation (sometimes deserved) of difficult work habits and lack of social skills. It therefore behooves companies to figure out ways to motivate and compensate the prickly but prolific lone inventor and to integrate the creative breakthroughs of such individuals. Organizations that desire a greater number of breakthroughs from lone inventors also must be willing to sort through and absorb a greater number of failures. That said, the increasingly popular managerial advice that collaboration will improve creativity appears to be correct, particularly for the typical manager who prefers to minimize any uncertainty in the innovation process. Still, if companies follow that advice and encourage all their inventors to collaborate, they ironically will also likely achieve fewer breakthroughs.

How Does Collaboration Influence Breakthroughs?

That raises the next question: What exactly is the role of collaboration? First, managers need to understand certain nuances about joint work, specifically, the difference between brokered and cohesive collaborations. Brokerage occurs when a single individual is the hub through which all collaborators interact. The opposite structure of cohesion occurs when collaborators develop separate and independent relationships with one another that do not include a central individual. (see “Two Types of Collaborations.”)

By brokering others, hub inventors gain first access to and control of information, enabling them to generate a greater number of new combinations. Yet this same social networking structure also makes it inherently more difficult for others to understand a focal inventor’s idea in order to critique, transfer and evolve it. Hence, brokers tend to generate more new combinations, but those innovations are less likely to be picked up and developed by others.9

That result suggests a reinterpretation of the “not invented here” syndrome,10 in which a group fails to embrace a superior external technology despite managerial pressure to do so. This typically is cast as a problem on the receiving end, specifically, that the receiving group isn’t working hard enough to adopt the technology or that it sabotages (either actively or passively) the transfer. But if the external technology is from a brokered collaboration, then the problem might very well lie with the transmitting end, specifically, that the hub inventor might lack the motivation, resources or time to transfer the innovation. And because that individual might be the only source with a complete understanding of the technology, the organization must depend on that person to participate in the transfer.

Brokered and cohesive collaborations have their relative pros and cons, and companies need to understand the various trade-offs. Manipulating the brokerage-cohesion variable can increase the generation of ideas or it can improve the diffusion of ideas, but it can’t accomplish both without additional help. In other words, managers must rely on other mechanisms (such as financial or professional incentives) to exploit the benefits and avoid the downsides of either approach. Instead of rewarding just the raw generation of ideas or patents by brokers, for example, companies also might consider giving sufficient credit to the transfer and adoption of knowledge and technology.

Neither brokered nor cohesive collaboration is inherently superior to the other; much depends on the organizational culture and the specific environment of the inventors. For example, although brokerage is better for the generation of new combinations, cohesion confers strong marginal benefits in collaborations that lack trust or involve fresh information. People who have recently moved from another organization thus would be better off collaborating within a cohesive structure in order to gain their new colleagues’ trust and transfer their external information most effectively.

Another important factor in collaborative work is the role of gatekeepers, people who span the boundaries between different groups.11 Gatekeepers tend to increase a company’s inventive creativity through their adoption of others’ new combinations. But, in addition to simply transferring ideas from one group to another, gatekeepers are more likely to invent new ideas. It is important to remember that not all gatekeepers are created equal. Companies should note that spanning a technological domain doubles the likelihood of generating new combinations in comparison with spanning an organizational boundary. In other words, even though technological and social borders tend to correlate, the creative opportunities appear to be twice as great at the technological boundaries.

Surprisingly, being embedded in a large and extended social network has little effect on the creativity of the individual inventor. But such networks do facilitate the diffusion of technology. This can be a disadvantage for companies that want to keep their internal technology from leaking to the outside world.12 That vulnerability is particularly acute for firms working with modular components to generate new combinations. Such relatively straightforward knowledge tends to diffuse easily, in that any inventor can recreate and build upon a previous inventor’s work. In contrast, very complex knowledge (such as a biotech manufacturing process that is not fully understood) diffuses only with great difficulty, in that no person, even one who works for the same firm and has access to the original source of the necessary knowledge, can recreate and build upon the work.13

The bottom line is that the collaborative structure of a lab has a powerful effect on its inventive output and chances of a breakthrough. Companies must realize how a structure that increases the likelihood of a breakthrough will also disrupt their incremental invention and efficiency. At one end of the spectrum, the presence of just solitary inventors increases the chances of a breakthrough (that is, higher maximum scores). At the other end, having only cohesive collaboration improves the odds for incremental and efficient invention and development (that is, a higher average score).

Does Diversity Help or Hurt?

Many companies have noticed that diversity and breakthroughs seem to co-occur. Yet that goes against the advantages of specialization and focus. This apparent contradiction can be resolved by considering the histogram of inventive creativity. Diversity helps generate more shots on goal although, on average, those shots are less successful. But diversity also gives rise to new and unexplored combinations that increase the probability of a highly skewed breakthrough.

Perhaps the most popular prescription for creativity and breakthroughs is multidisciplinary collaboration.14 According to its proponents, breakthroughs require the juxtaposition of inventors with differing expertise. Xerox Corp.’s Palo Alto Research Center is often held up as the exemplar here. During its heyday, Xerox PARC employed a phenomenally diverse assortment of natural and social scientists, engineers and artistic personnel. Yet the evidence linking breakthroughs with multidisciplinary collaborations remains mixed. The truth is that, on average, it’s more productive to search within established disciplines.15 Or, when trying to cross-pollinate between fields, the more productive approach is to combine areas that have some common ground. Consider the rapid emergence of nanotechnology, which relies on know-how from two fields — semiconductor manufacturing and mechanical engineering — that have deep foundations in the physical sciences.

My own research studying more than 17,000 patents has found that the greater the divergence between collaborators’ fields of expertise, the lower the overall quality of their output. But multidisciplinary collaboration increases the variance of the outcome, such that failures as well as breakthroughs are more likely. (See “The Effect of Multidisciplinary Collaboration.”) Case in point: the exciting research in the area of behavioral economics, which resides at the intersection of two disparate fields — psychology and economics.

Companies should go for depth, not breadth, when assembling a diverse team. In general, individuals with deep expertise will be more likely to see potential synergies across fields than will those with broader but shallower knowledge. Take, for example, the laboratory headed by Robert S. Langer, an engineering professor at the Massachusetts Institute of Technology, who is one of the most productive inventors alive today (as measured by U.S. patents). Langer’s own expertise is in chemical engineering, and he has staffed his lab with PhDs from a range of disciplines. The result is a veritable innovation factory that has been responsible for nearly 800 papers, 500 patents and a dozen very successful start-ups.

What’s the Role of Science?

Most inventors search locally, in the sense that they typically alter only one part of a system at a time, either replacing it or reconfiguring it relative to other components.16 Such strategies can be very successful because they rely on accumulated knowledge and previous successes. The downside, however, is that searching locally typically generates only incremental improvements and few breakthroughs. Furthermore, the process breaks down completely when the components are interdependent, that is, when a small change in one part can lead to a huge difference in the overall system. In that way, technology is like a landscape in which systems of interdependent components create a complex and rugged terrain where local search or simple “hill-climbing” algorithms quickly become trapped on local maxima (that is, incremental innovations) instead of global peaks (that is, breakthroughs).17

Interdependent components might be difficult to work with, but they are also the stuff of many breakthroughs.18 With semiconductors, minute impurities of dopant (in parts per million) can result in either a transistor — the basic building block of the information revolution — or a relatively useless resistor. As components become more interdependent, though, people become less effective in struggling with the increasing complexity, and they have greater difficulty predicting how their inventions will work. Thus the overall effect of complex interdependence is similar to that of collaborative diversity: The typical outcome is less fruitful, but the possibility of a breakthrough increases.

So, then, what’s the role of science? My fieldwork, research and experience suggest that the scientific method and knowledge help provide a useful map of the technology landscape.19 Armed with such information, inventors can more efficiently find the optimal combinations of components. By explaining why phenomena occur, science enables (1) predictions of how certain combinations will work; (2) insights into which combinatorial spaces can be winnowed because they lack potential; (3) simulations of fertile possibilities; and (4) theoretical encouragement in the face of empirical failure. As such, the application of science is particularly powerful when an inventor is working with interdependent components for which little empirical knowledge has been developed (and, indeed, for which the development of empirical knowledge would be prohibitively expensive).

In essence, the application of scientific knowledge helps inventors exploit the interdependencies among technological components. Such interdependencies provide the potential for breakthroughs, but they can substantially complicate the search process. In effect, scientific methods make the process of invention less random, enabling inventors to make fewer but better shots on goal. On the other hand, science by itself will also decrease the variability of outcomes because it often discourages the pursuit of unconventional ideas that lead to technological dead ends as well as novel discoveries. But when science is applied to complex and interdependent components, it can unlock the potential for breakthroughs.

Managers should be aware that science facilitates the diffusion of an invention20 and hence makes it more difficult for companies to capture the full economic fruits of their investments. Moreover, developing scientific capabilities is neither cheap nor easy. It requires a long-term investment, the motivation of professionals (who often owe greater allegiance to their professions and communities than to their companies) and a conscious effort to generate a financial return from an activity and institution not originally designed with pecuniary goals. Furthermore, companies should remember that science generates the greatest benefits when it is closely integrated with other activities within the organization.

Managerial Challenges

Different factors can dramatically affect a company’s inventive output, with each having a different impact on the three measures of inventive success: shots on goal, average score and maximum scores (See “The Levers of Invention.”) Consequently, companies first need to identify how they want to improve their innovation process and then take the appropriate measures. If an organization has an adequate number of shots on goal but a poor average score, for example, it might do well to consider a substantial investment in science and basic research. If, on the other hand, the primary problem is a paucity of shots on goal, the company might be better off focusing on cultivating technology brokers and increasing the diversity in its labs.

Of course, the creativity process always will rely to some degree on serendipity — the chance meeting between two researchers that sparks some creative leap in thinking. The crucial thing to remember, though, is that companies do have tremendous control over their innovation processes, enabling them to address deficiencies in certain areas. Indeed, by understanding the histogram of inventive creativity and managing the factors that shape that distribution, a firm can greatly improve its capacity to innovate in specific ways that make the best sense for the organization as a whole.