What People Want (and How to Predict It)

The Year 2007 was a terrible year for many big movie stars. One major exception was Will Smith, whose film “I Am Legend” set a box-office record for a movie opening in December, taking in $77 million. In 2008, Smith’s star vehicle “Hancock” grossed more than $625 million worldwide despite poor critical reviews. Smith’s success was not all that surprising, however: With the exception of the Harry Potter movies, those in which Smith star have higher opening weekends and average box-office receipts than movies with any other male lead.1

Does Smith know something that Jim Carrey and others do not? Quite possibly: When Smith went to Hollywood to start his film career, he and his business manager studied a list of the 10 top-grossing movies of all time. “We looked at them and said, OK, what are the patterns?” Smith recalls. “We realized that 10 out of 10 had special effects. Nine out of 10 had special effects with creatures. Eight out of 10 had special effects with creatures and a love story.”2

Smith calls himself a “student of universal patterns” and studies box-office results after every weekend, looking for patterns of success. Given his track record of choosing films that reliably deliver $120 million or more, he is clearly an astute observer.

Smith’s ability to analyze and predict which movies are likely to succeed belies conventional wisdom on predicting consumer taste. Such predictions are viewed as an art, not a science. The reasons for success or failure are inscrutable. Producers of movies, music, books and apparel pursue their artistic visions and offer them to the public, which may or may not recognize genius when it sees it.

It’s easy to see why most people view the prediction of taste as an art. Historically, neither the creators nor the distributors of “cultural products” have used analytics — data, statistics, predictive modeling — to determine the likely success of their offerings. Instead, companies relied on the brilliance of tastemakers to predict and shape what people would buy. If Coco Chanel said hemlines were going up, they did. Feelings, not data, were critical. Harry Cohn, the founder of Columbia Pictures, believed he could predict how successful a movie would be based on whether his backside squirmed as he watched (if it did, the movie was no good).

Such tastemakers still exist. Wines that receive a 90+ score from Wine Spectator are virtually guaranteed high market demand. Manufacturers of everything from automobiles to toasters rely on the Color Association of the United States’ recommendations to determine color trends for their products. The success of Columbia Records’ cohead, Rick Rubin, has been attributed in part to “the simultaneously mystical and entirely decisive way he listens to a song.”3

Image courtesy of Flickr user jacilluch.

Creative judgment and expertise will always play a vital role in the creation, shaping and marketing of cultural products. But the balance between art and science is shifting. Today companies have unprecedented access to data and sophisticated technology that allows even the best-known experts to weigh factors and consider evidence that was unobtainable just a few years ago.

As a result, the prediction of consumer taste is quietly becoming a prominent feature of the entertainment and shopping landscape. Creators and distributors of cultural products are attempting to predict how successful a particular product will be before, during or after its creation. Consumers of cultural products can draw upon recommendations — a form of prediction as well — about which products or product attributes will appeal to them.

In this article we describe the results of a study of prediction and recommendation efforts for a variety of cultural products. (See “About the Research.”) We explain why prediction and recommendation technologies are important, the different approaches used to make predictions, the contexts in which these predictions are applied and the barriers to more extensive use.

If the success and appeal of cultural products can be predicted, why not any other product or service? For executives leading any company whose main offerings are consumer products, such knowledge will be increasingly critical to success. The sophisticated prediction of consumer tastes will help guide investment decisions for virtually all consumer products and services. Today it is already common for consumers to consult online comments and ratings, and both manufacturers (Dell, Lego, Intuit, Timberland) and retailers (Costco, Sears, Macy’s) make available such opinions. As offerings proliferate and consumers’ “share of mind” comes under assault from a bombardment of choices and opinions, recommendation technologies will allow consumers to evaluate options and synthesize ratings more systematically. Prediction will be equally useful for creators of products and content. Just as a consumer products company wouldn’t dream of launching a new product without testing it with consumers first, no company will launch any expensive-to-create product or content offering without subjecting it to some form of systematic prediction or test. The earlier in the development cycle the predictions can be made, the more useful they will be.

Prediction Technologies Come of Age

Tools designed to predict and shape what consumers want have been around for decades. But as with so many information technologies, they did not begin to take off until the 1990s.

In the 1930s and 1940s, George Gallup attempted, with little success, to persuade Hollywood to apply his newly developed public opinion polls to discover viewers’ tastes.4 In the early 1940s, the Bureau of Applied Social Research at Columbia University (previously known as the Office of Radio Research) developed the Lazarsfeld-Stanton Program Analyzer, which required subjects to record positive and negative reactions to movies as they watched them.5 One of the earliest examples of hit prediction software in the film industry, ERIS, dates to the 1970s.6 Doubts persisted, however; the screenwriter William Goldman famously noted in his 1983 book Adventures in the Screen Trade that “nobody knows anything” about the factors associated with the commercial success of a movie. While strides have been made in the use of prediction for producers and distributors, more progress has been made on the consumer recommendation front.

Efforts to produce useful recommendations for consumers began to come to fruition in the late 1990s, when Amazon.com Inc. pioneered the widespread commercial use of predictions with “collaborative filtering.” This software made recommendations by analyzing a consumer’s past choices and making correlations with other products that he or she might like. Collaborative filtering can be useful in pointing shoppers toward products they hadn’t known existed, but it is also limited. For example, it has no way of knowing when someone has purchased an item for someone else and would have no interest in other products related to that single purchase.

More recently, the online movie distributor Netflix Inc. has had success with another form of collaborative filtering. Its software produces movie recommendations by correlating a data set of more than a billion movie ratings from its customers. Another example, the TiVo Suggestions feature, selects shows it predicts consumers will like based on their viewing patterns and ratings of other programs, using a combination of techniques.7

Amazon and Netflix are primarily distributors of cultural products; their recommendation systems are an adjunct to their main business model. Companies that specialize in the recommendation process itself have also emerged in recent years. ChoiceStream Inc. develops recommendation software for movies, television, books and consumer goods, and licenses its software to distributors of these products. Media Predict Inc. has created prediction markets for movies, books, music and television. The company partnered with Touchstone Books, a Simon & Schuster imprint, to use a prediction market in 2007 to select one book to publish based on rankings in a prediction market. The book selected, Hollywood Car Wash, was a moderate commercial success.8 Other companies focus on particular media or product niches. The Echo Nest Corp. and Platinum Blue Music Intelligence provide music recommendation capabilities for online music distributors.

While recommendation technologies began in the United States, they are spreading around the world. Acquamedia Technologies SI, a Spanish company, produces recommendation software for music sold over mobile telephone networks. Silver Egg Technology Co., a Japanese company, provides software to help Japanese online retailers recommend products to their customers.

Predictions of what products will be successful for creators and distributors of cultural content are less common. It is easiest after the product has been developed, when its attributes are clear and there are some indicators of its popularity. For example, a movie studio’s home video distribution business makes predictions (primarily using regression analysis) of how many copies to produce, and they are usually fairly accurate. Their predictions before the movie is actually made, however, are often wildly inaccurate.

Despite the difficulties of prediction before creation, U.K.-based Epagogix Ltd. makes predictions of movie success based on script attributes. For example, as part of a test for a hedge fund, it predicted that the 2007 film “Lucky You” would bomb, bringing in only $7 million at the box office. The film, which featured a major star (Drew Barrymore), a well-known director and screenwriter, and a plot about a popular topic, professional poker, cost $50 million to make. Epagogix, however, was on target, as the film brought in a paltry $6 million.

Valuing Prediction and Recommendation

One of the reasons that recommendation offerings are proliferating is that consumers today are overwhelmed by “the paradox of choice” — so many choices to make, and no easy way to distinguish among the offerings. Producers face the opposite problem: They need to make wise investment decisions in a world cluttered with cultural products. They seek to mitigate the increasing risks of developing and distributing new offerings. For both consumers and producers, prediction and recommendation capabilities are particularly important today.

Consider the dilemma faced by consumers trying to “keep up.” They likely agree with the sentiment recently expressed by a New York Times media critic: “Like most Americans, I am overwhelmed by the velocity of everyday life and the volume of media that goes with it.”9 With so many options and time at a premium, consumers need help deciding what media they are most likely to enjoy.

The number of book titles published in the United States, for example, grew by more than 50% during the 10-year period from 1994 to 2004. Other countries are also publishing record numbers of books a year. But according to the Book Industry Study Group Inc., of the almost 300,000 books published in the United States in 2004, fewer than a quarter of them sold more than 100 copies.10 Despite these increases in book production, surveys indicate that Americans are reading less each year.11 Clearly, both book publishers and readers are in a bind.

Movie studios around the world are churning out more movies than viewers can watch. The number of Hollywood films released in 2006 was 607 — an 11% increase over the previous year, and an all-time high. That total was almost double the number released in 1990, yet few have the time to see twice as many films as they did just a couple of decades ago.12 Indian film production companies are even more prolific, releasing more than 1,000 new feature-length movies a year. And books and movies are just the tip of the iceberg, as people increasingly spend their time watching professional and amateur “cultural production” on sites like YouTube via their laptops, mobile phones or PDAs.

This trend of increasing production takes place at a time when at least some cultural products are more expensive to create. Studio movies, in particular, require big bets. According to the Motion Picture Association of America, the combined average cost of making and marketing a studio picture in 2006 was $100.3 million.13 And most films are not successful commercially; one economist estimates that 6% of films accounted for 80% of the industry’s profits over the past decade; 78% of movies lost money over that period.14 According to one industry report, these economics are taking a toll on studios’ profits. In aggregate, the 132 movies released in 2006 by the major movie studios are expected to lose $1.9 billion after their five-year cycle of theatrical release, DVD sales, television deals and all additional sources of income.15

The increased production and financial drain creates a greater need for both predictions and recommendations. Producers need to create products with a greater likelihood of success. And both producers and consumers have an interest in connecting consumers with cultural content they will like, and hence keep buying.

Technology — Prediction’s Great Enabler

A key reason why prediction and recommendation are important now is that they are easier to realize, from a technical standpoint. Relatively new distribution channels, including the Internet for movies and books, and mobile phones for music, can be embedded with software that provides recommendations in the distribution process. These channels also generate detailed data on customer behavior and preferences. Of course, while these channels can provide a great deal of information about products, there is usually not enough bandwidth or time available for consumers to make truly effective choices. And the smaller the aperture to the customer (a mobile phone screen, for example), the more important it is to assist the customer in making choices, because the amount of information able to be displayed at one time is so limited.

The best reason to use recommendations, however, is that they seem to work — at least for consumers (predictions of success for creators are too new to judge effectiveness). Netflix, for example, has found that customers like its recommendations about 10% better (half a star in their five-star rating system) than their own selections. O2 PLC, a U.K.-based mobile-network operator, found that 97% of its customers opted to use a service to predict and present mobile content that matches their tastes. The Hollywood Stock Exchange aggregates virtual bets from several hundred thousand players on which movies, stars and directors will prosper or decline. A high total of bets in a simulated currency indicates a prediction of success, and few or low bets a failure. One study found that the Exchange’s prerelease predictions of a movie’s box-office take were quite accurate, and comparable to the best expert predictions.16

Several companies have also discovered that their recommendations help to sell more products. Acquamedia found that revenue for its mobile phone network customers increased between 15% and 20% when consumers made use of its music recommendations. Silver Egg reported double-digit growth when its customers were offered recommendations for media purchases. Blockbuster Inc. has seen decreased customer churn month to month since it deployed the ChoiceStream Inc. recommendation engine. Overstock.com Inc. employed a ChoiceStream-based Gift Finder on its Web site in time for the 2006 holiday season, and the technology increased revenue by 250% from those who used it.17 Overstock also found that in the first 18 months after launching a refined e-mail targeting system, e-mail marketing revenue doubled and the average order size increased 5.9%.18

An Array of Techniques and Technologies

Executives who want to incorporate predictive technologies in their businesses must first understand the variety of approaches that already exist. (See ”The Prediction Lover’s Handbook,” page 32.) The first generation of technology, collaborative filtering, makes correlations either item by item or customer by customer. This approach is still employed today — not only by Amazon and Netflix, but also by companies such as mobile telephone music recommender LiveWire Mobile Inc., which distributes musical choices through more than 20 wireless carriers around the world.

A relatively new approach to recommendation focuses on the attributes of an item. A movie, for example, might be classified by its length, genre (“criminal thriller”), theme (“unlikely criminals”), tone (“ominous, forceful, gritty and tense”), critic’s rating and so forth. An analysis of the movies a customer likes could lead to recommendations of other movies with similar attributes. ChoiceStream does this for both movies and online shopping. The online radio station Pandora (using classifications created by its employees) and the music software recommendation company Echo Nest (using computational analysis of sound as well as textual analysis of online content about music) have classified different aspects of thousands of songs — including timbre, key, tempo, time signature and instruments.

Other possible approaches to prediction include markets like the Hollywood Stock Exchange or Media Predict. Platinum Blue Music Intelligence employs “spectral deconvolution” of sound waves to identify songs that would be appealing to a particular listener. Epagogix uses a proprietary expert system with neural network-based algorithms to predict a movie’s success before it’s made, and many studios use regression analysis to project the success of a film before release.

Some companies are beginning to add social networking as a means of recommending cultural products. If your friends like certain songs and movies, perhaps you will like them, too — and if you and a stranger like the same songs and movies, perhaps you should become friends. LiveWire Mobile and Last.fm Ltd. have a social networking element in their music offerings, and Netflix has a “Friends” service that lets customers share movie preferences and reviews with a community.

Each prediction or recommendation approach has particular strengths and weaknesses in the context of the application. Collaborative filtering, for example, requires a substantial amount of data on past purchases to work effectively. Even when enough data exists, some experts believe collaborative filtering reduces the diversity in purchases made and makes blockbuster hits even bigger.19 Neural networks also require a large amount of data. Attribute-based recommendation requires that someone classify cultural products according to several key attributes; if there isn’t already a source of attributes for a product, developing one can be difficult. Prediction markets need a large number of independent participants to succeed; most offer some sort of prize or token reward to attract them. Of course, if a third party has already rounded up all the necessary resources to offer predictions or recommendations for your product, all you have to do is pay for them.

The best recommendation tools perform a balancing act: They connect to consumers’ sense of individuality as well as their group identification. Similarly, the tools must come up with recommendations that stretch horizons with suggestions that are new and a bit surprising, yet not off-putting. Recommendation approaches vary in how much access to the “long tail” of niche or obscure products they provide. Most recommendation engines offer a balance of the familiar and the unexplored.

At LiveWire Mobile, for example, customers want both reliable and well-known songs similar to those they like, as well as songs from different parts of the world and from different musical genres that may challenge or advance their tastes. But LiveWire Mobile’s business model is a pay-per-song model, which makes its customers somewhat more conservative than they might be in a subscription model. The lesson for executives is that if people are buying your product one at a time, choose a recommendation approach that provides conservative recommendations; if they like you enough to pay you a monthly fee, they’re probably open to a recommendation engine that provides pleasant surprises.

Finally, because markets for cultural products shift over time, it is critical to monitor changing market conditions continuously to identify emerging trends. “Model management” is essential to the development of recommendation algorithms that reflect lessons from experience, test assumptions and improve the accuracy of predictions. Netflix, for example, developed many of its recommendation approaches with customers who were Internet pioneers; now that it’s also serving later adopters, the company’s analysts feel the need to develop new tests and algorithms.

On the cutting edge of technology are attempts to identify patterns of intrinsic appeal to human viewers or, more commonly, listeners. Scientists are learning more about the mathematical connections hidden in music and how they contribute to a desire to hear certain songs repeatedly — a condition known as “earworms” or “cognitive itch.”20 Platinum Blue Music Intelligence has applied this research to analyze a song and provide recommendations to increase the likelihood that the song will be a hit — by fine-tuning the bass line, for example. CEO Mike McCready describes his company’s goal as “helping both artists and producers by explaining factors that increase the odds of a successful release.”

The company’s analysis has resulted in the creation of 60 distinct clusters, a dozen or so of which are active at any particular point in history. A Chopin prelude may be in the same cluster as songs by Frank Sinatra, Genesis and ZZ Top. Billed as a tool to assist artists and producers, Platinum Blue’s technology uses spectral sound-wave analysis to offer advice. For example, the tool was used to analyze the song “Crazy,” by Gnarls Barkley. The analysis found that “Crazy” belonged to the same hit cluster as several recent hit songs as well as older hits by Olivia Newton-John and Mariah Carey. The data clearly indicated that “Crazy” was going to be a huge success, which it was.

New technologies will continue to emerge for analyzing and predicting consumer tastes. Innerscope Research is beginning to employ biological approaches to study consumer engagement for advertising and television programs.21 The company measures biological indicators of mental engagement, such as heart rate and galvanic skin response. NASA developed an even more direct measure of human attention using brain waves, but thus far the technology has not been successfully applied commercially. As soon as it’s clear that money can be made using these biological assessment tools, their use will undoubtedly grow despite some observers’ moral and ethical qualms.

Predictions and the Creative Process

In the great majority of cases we studied, recommendations were made after the cultural product was created and assisted the customer in choosing among finished offerings. Prediction can also be used, as in the European movie theater chain Kinepolis Group NV, to project the staffing levels needed in a particular theater for a particular film on a particular weekend. Studios use similar prediction approaches to judge how many DVDs to manufacture and ship. It is also possible to use the content of recommendations even before and during product creation.

This precreation approach is most actively explored in the movie business, where long production cycles and high costs predominate. Movie producers have long had rules of thumb to guide their moviemaking decisions — a star brings in the crowds, audiences like happy endings, PG-13 rated films are moneymakers, sequels make between two-thirds and three-quarters of the original movie’s box office and so forth. Studio executives at two studios told us that these maxims are widely believed within their companies, though there are plenty of well-known exceptions to these rules. And financial executives at many studios make predictions of how much money a particular picture will make — both before and after initial theatrical release (in so-called decay models). It is clear from the high percentage of unsuccessful films that the decision process is largely nonscientific. One reason is that individual studios produce relatively few movies and cannot compile extensive data. One studio executive, for example, told us that since the studio produces an average of only eight movies per year, a highly statistical approach to predicting success and failure would be impossible.

However, scientific help is now available for moviemakers before their work is done. Epagogix, in particular, is focused on predicting the likelihood of success of movie scripts before production even begins. The company’s neural network analyses identify attributes of scripts that are correlated with either success or failure as defined by box-office revenue. This would seem to be an appealing prospect to studios, and by most accounts it is technically possible.

Using the simple metric of whether a film will recoup its production costs at the U.S. box office, Epagogix is able to predict “turkeys” and “eagles” twice as accurately as studios. It can also make specific recommendations that it predicts will increase box-office receipts: For one film, the software recommended reducing the number of scene locations, a decision that would not only increase the likely box-office take, but also significantly reduce production costs. Hedge fund managers have discussed with Epagogix a studio partnership that would predict the success of a film before it is made, and well-known agents have discussed using Epagogix’s tools before their actor-clients accept roles (particularly those who are paid in part as a percentage of box-office receipts).

Some studio executives themselves, however, have thus far been less than enthusiastic about turning their decision-making art into a science. The primary obstacles appear to be cultural rather than analytical or technological. One executive suggested to Epagogix executives that he would be ostracized by the Hollywood community — and not invited to the good parties! — if word got out that he was producing movies based on analytical prediction models. Epagogix has also developed other contexts in which its prediction approaches might be useful, including typical business situations like “mak[ing] the best objective decisions about spending risk capital and managing operational budgets.”22

This resistance to science has historical precedence in Hollywood, as noted as early as 1941 by Leo Rosten, the academic, screenwriter and humorist: “The movie makers work with hunches, not logic; they trade in impressions rather than analyses. It is natural that they court the intuitive and shun the systematic, for they are expert in the one and untutored in the other.”23 A financial executive at a major studio confirmed in an interview that so far all prediction models have made relatively little headway with executives who make film production decisions, though he is hoping that they will be applied more frequently in the future.

An HBO Inc. executive was similarly skeptical of the feasibility of using analytics on the creative side of the company’s business. HBO’s executives view their role as “human curators” whose discerning audience seeks high-quality original programming that confounds conventional expectations. HBO does employ some analytics, but not for prediction. Its production planning department, for example, uses software with codified rules such as “R movies must not be scheduled in the daytime” to help schedule programming.

It is the artists themselves who may ultimately embrace the use of prediction techniques to guide their decision making for everything from picking movie scripts to fine-tuning a song to optimize its market potential. At Platinum Blue Music Intelligence, the response has been mixed, according to the CEO. “We got thousands of e-mails from musicians essentially saying either, ‘This is just another example of the Man trying to keep us down using impersonal computers’ or ‘When can I get this technology to help me get discovered?’” Those who adopt the technologies may well find they have a powerful new tool to help them.

Business Model Risks and Opportunities

Those who want to incorporate the prediction or recommendation of cultural products into their businesses need to consider several management issues. One is the business model they choose to adopt. Should prediction be the only way to make money, or must it accompany a business model that makes money through other approaches as well?

Many of the companies we studied, including Apple, Netflix, LiveWire Mobile and Amazon, make their money primarily by being distributors of cultural products, not recommenders. The recommendation work they do is a small adjunct to their distribution business. And if the distribution model is problematic — as in the case of Pandora, where the need to pay royalties to record companies for online music almost shut down the company recently — the recommendations alone may not be enough to let an organization thrive. When we suggested to Netflix CEO Reed Hastings that the company’s recommendation capabilities might be sold to other online or telecom-based movie distributors, he replied that recommendations alone were insufficiently valued by most online distributors.

Most of the companies we encountered that provide only recommendation or prediction capabilities are relatively small. To thrive, they need to spread their recommendation capabilities across a variety of industries. ChoiceStream, for example, now offers recommendations not only for movies and online retail, but also for books (through Borders.com), TV listings and music. The company is considering using its technology broadly for targeted online advertising, and Overstock already uses it for this purpose. ATG Recommendations (formerly CleverSet), a recent startup in the recommendation engine business, has customers who are using its tools to sell wine, baked goods, T-shirts and software through the Internet. The Hollywood Stock Exchange has provided the model and tools for online markets in data storage, drug development in pharmaceuticals, and predictions for Popular Science magazine. Recommendation software providers say that their approaches rapidly build up a font of rich, accurate consumer preference data, which they believe to be potentially valuable to producers of the products and services they recommend. Some of these small firms will not succeed, however; the recommendations company MatchMine offered a “portable” set of recommendations that could be transferred across different Web sites, but it recently discontinued operations.

The need to continually update and refine models is another management issue. Netflix offers incentives to external analysts to improve its model through the Netflix Prize: $1 million to anyone who can improve the company’s prediction algorithm by 10% (one group is almost there at about 9.5%, but the contest is already more than two years old). Amazon continues to refine its collaborative filtering model. Attribute-based recommendation companies such as ChoiceStream, which serves multiple industries and customers, must refine not only their analytical models, but also their means of collecting attributes economically. And firms like Echo Nest whose sole offering is their recommendation engine must find ways to make their business profitable through partnering, online advertising or other means.

Finally, despite the great promise of prediction and recommendation systems, it’s important for executives to avoid going to the extreme. These systems are not a substitute for decision making, nor do they provide automatic, infallible answers. Using these tools does not obviate the need for business judgment or cultural acumen. As one movie studio executive put it, “Consumers already complain they are being pandered to. Isn’t this the ultimate in pandering?” Even Will Smith doesn’t rely solely on his analysis in choosing scripts; he also seeks input from his family and friends. Creating successful cultural products will always be a mixture of art and science. It appears, however, that the amount of science in the mixture is increasing. It’s likely that the same science-based approaches to predicting success and recommending products will continue to transform not just the cultural products industry, but any other industry in which new products are expensive and risky, and in which customers lack the time and attention to understand differences among proliferating offerings.