The Year 2007 was a terrible year for many big movie stars. One major exception was Will Smith, whose film “I Am Legend” set a box-office record for a movie opening in December, taking in $77 million. In 2008, Smith’s star vehicle “Hancock” grossed more than $625 million worldwide despite poor critical reviews. Smith’s success was not all that surprising, however: With the exception of the Harry Potter movies, those in which Smith star have higher opening weekends and average box-office receipts than movies with any other male lead.1

Does Smith know something that Jim Carrey and others do not? Quite possibly: When Smith went to Hollywood to start his film career, he and his business manager studied a list of the 10 top-grossing movies of all time. “We looked at them and said, OK, what are the patterns?” Smith recalls. “We realized that 10 out of 10 had special effects. Nine out of 10 had special effects with creatures. Eight out of 10 had special effects with creatures and a love story.”2

Smith calls himself a “student of universal patterns” and studies box-office results after every weekend, looking for patterns of success. Given his track record of choosing films that reliably deliver $120 million or more, he is clearly an astute observer.

Smith’s ability to analyze and predict which movies are likely to succeed belies conventional wisdom on predicting consumer taste. Such predictions are viewed as an art, not a science. The reasons for success or failure are inscrutable. Producers of movies, music, books and apparel pursue their artistic visions and offer them to the public, which may or may not recognize genius when it sees it.

It’s easy to see why most people view the prediction of taste as an art. Historically, neither the creators nor the distributors of “cultural products” have used analytics — data, statistics, predictive modeling — to determine the likely success of their offerings. Instead, companies relied on the brilliance of tastemakers to predict and shape what people would buy. If Coco Chanel said hemlines were going up, they did. Feelings, not data, were critical. Harry Cohn, the founder of Columbia Pictures, believed he could predict how successful a movie would be based on whether his backside squirmed as he watched (if it did, the movie was no good).

Such tastemakers still exist. Wines that receive a 90+ score from Wine Spectator are virtually guaranteed high market demand. Manufacturers of everything from automobiles to toasters rely on the Color Association of the United States’ recommendations to determine color trends for their products. The success of Columbia Records’ cohead, Rick Rubin, has been attributed in part to “the simultaneously mystical and entirely decisive way he listens to a song.”3

Creative judgment and expertise will always play a vital role in the creation, shaping and marketing of cultural products. But the balance between art and science is shifting. Today companies have unprecedented access to data and sophisticated technology that allows even the best-known experts to weigh factors and consider evidence that was unobtainable just a few years ago.

As a result, the prediction of consumer taste is quietly becoming a prominent feature of the entertainment and shopping landscape. Creators and distributors of cultural products are attempting to predict how successful a particular product will be before, during or after its creation. Consumers of cultural products can draw upon recommendations — a form of prediction as well — about which products or product attributes will appeal to them.

In this article we describe the results of a study of prediction and recommendation efforts for a variety of cultural products. (See “About the Research.”) We explain why prediction and recommendation technologies are important, the different approaches used to make predictions, the contexts in which these predictions are applied and the barriers to more extensive use.

If the success and appeal of cultural products can be predicted, why not any other product or service? For executives leading any company whose main offerings are consumer products, such knowledge will be increasingly critical to success. The sophisticated prediction of consumer tastes will help guide investment decisions for virtually all consumer products and services. Today it is already common for consumers to consult online comments and ratings, and both manufacturers (Dell, Lego, Intuit, Timberland) and retailers (Costco, Sears, Macy’s) make available such opinions. As offerings proliferate and consumers’ “share of mind” comes under assault from a bombardment of choices and opinions, recommendation technologies will allow consumers to evaluate options and synthesize ratings more systematically. Prediction will be equally useful for creators of products and content. Just as a consumer products company wouldn’t dream of launching a new product without testing it with consumers first, no company will launch any expensive-to-create product or content offering without subjecting it to some form of systematic prediction or test. The earlier in the development cycle the predictions can be made, the more useful they will be.

Prediction Technologies Come of Age

Tools designed to predict and shape what consumers want have been around for decades. But as with so many information technologies, they did not begin to take off until the 1990s.

In the 1930s and 1940s, George Gallup attempted, with little success, to persuade Hollywood to apply his newly developed public opinion polls to discover viewers’ tastes.4 In the early 1940s, the Bureau of Applied Social Research at Columbia University (previously known as the Office of Radio Research) developed the Lazarsfeld-Stanton Program Analyzer, which required subjects to record positive and negative reactions to movies as they watched them.5 One of the earliest examples of hit prediction software in the film industry, ERIS, dates to the 1970s.6 Doubts persisted, however; the screenwriter William Goldman famously noted in his 1983 book Adventures in the Screen Trade that “nobody knows anything” about the factors associated with the commercial success of a movie. While strides have been made in the use of prediction for producers and distributors, more progress has been made on the consumer recommendation front.

Efforts to produce useful recommendations for consumers began to come to fruition in the late 1990s, when Amazon.com Inc. pioneered the widespread commercial use of predictions with “collaborative filtering.” This software made recommendations by analyzing a consumer’s past choices and making correlations with other products that he or she might like. Collaborative filtering can be useful in pointing shoppers toward products they hadn’t known existed, but it is also limited. For example, it has no way of knowing when someone has purchased an item for someone else and would have no interest in other products related to that single purchase.

More recently, the online movie distributor Netflix Inc. has had success with another form of collaborative filtering. Its software produces movie recommendations by correlating a data set of more than a billion movie ratings from its customers. Another example, the TiVo Suggestions feature, selects shows it predicts consumers will like based on their viewing patterns and ratings of other programs, using a combination of techniques.7

Amazon and Netflix are primarily distributors of cultural products; their recommendation systems are an adjunct to their main business model. Companies that specialize in the recommendation process itself have also emerged in recent years. ChoiceStream Inc. develops recommendation software for movies, television, books and consumer goods, and licenses its software to distributors of these products. Media Predict Inc. has created prediction markets for movies, books, music and television. The company partnered with Touchstone Books, a Simon & Schuster imprint, to use a prediction market in 2007 to select one book to publish based on rankings in a prediction market. The book selected, Hollywood Car Wash, was a moderate commercial success.8 Other companies focus on particular media or product niches. The Echo Nest Corp. and Platinum Blue Music Intelligence provide music recommendation capabilities for online music distributors.

While recommendation technologies began in the United States, they are spreading around the world. Acquamedia Technologies SI, a Spanish company, produces recommendation software for music sold over mobile telephone networks. Silver Egg Technology Co., a Japanese company, provides software to help Japanese online retailers recommend products to their customers.

Predictions of what products will be successful for creators and distributors of cultural content are less common. It is easiest after the product has been developed, when its attributes are clear and there are some indicators of its popularity. For example, a movie studio’s home video distribution business makes predictions (primarily using regression analysis) of how many copies to produce, and they are usually fairly accurate. Their predictions before the movie is actually made, however, are often wildly inaccurate.

Despite the difficulties of prediction before creation, U.K.-based Epagogix Ltd. makes predictions of movie success based on script attributes. For example, as part of a test for a hedge fund, it predicted that the 2007 film “Lucky You” would bomb, bringing in only $7 million at the box office. The film, which featured a major star (Drew Barrymore), a well-known director and screenwriter, and a plot about a popular topic, professional poker, cost $50 million to make. Epagogix, however, was on target, as the film brought in a paltry $6 million.

Valuing Prediction and Recommendation

One of the reasons that recommendation offerings are proliferating is that consumers today are overwhelmed by “the paradox of choice” — so many choices to make, and no easy way to distinguish among the offerings. Producers face the opposite problem: They need to make wise investment decisions in a world cluttered with cultural products. They seek to mitigate the increasing risks of developing and distributing new offerings. For both consumers and producers, prediction and recommendation capabilities are particularly important today.

Consider the dilemma faced by consumers trying to “keep up.” They likely agree with the sentiment recently expressed by a New York Times media critic: “Like most Americans, I am overwhelmed by the velocity of everyday life and the volume of media that goes with it.”9 With so many options and time at a premium, consumers need help deciding what media they are most likely to enjoy.

The number of book titles published in the United States, for example, grew by more than 50% during the 10-year period from 1994 to 2004. Other countries are also publishing record numbers of books a year. But according to the Book Industry Study Group Inc., of the almost 300,000 books published in the United States in 2004, fewer than a quarter of them sold more than 100 copies.10 Despite these increases in book production, surveys indicate that Americans are reading less each year.11 Clearly, both book publishers and readers are in a bind.

Movie studios around the world are churning out more movies than viewers can watch. The number of Hollywood films released in 2006 was 607 — an 11% increase over the previous year, and an all-time high. That total was almost double the number released in 1990, yet few have the time to see twice as many films as they did just a couple of decades ago.12 Indian film production companies are even more prolific, releasing more than 1,000 new feature-length movies a year. And books and movies are just the tip of the iceberg, as people increasingly spend their time watching professional and amateur “cultural production” on sites like YouTube via their laptops, mobile phones or PDAs.

This trend of increasing production takes place at a time when at least some cultural products are more expensive to create. Studio movies, in particular, require big bets. According to the Motion Picture Association of America, the combined average cost of making and marketing a studio picture in 2006 was $100.3 million.13 And most films are not successful commercially; one economist estimates that 6% of films accounted for 80% of the industry’s profits over the past decade; 78% of movies lost money over that period.14 According to one industry report, these economics are taking a toll on studios’ profits. In aggregate, the 132 movies released in 2006 by the major movie studios are expected to lose $1.9 billion after their five-year cycle of theatrical release, DVD sales, television deals and all additional sources of income.15

The increased production and financial drain creates a greater need for both predictions and recommendations. Producers need to create products with a greater likelihood of success. And both producers and consumers have an interest in connecting consumers with cultural content they will like, and hence keep buying.

Technology — Prediction’s Great Enabler

A key reason why prediction and recommendation are important now is that they are easier to realize, from a technical standpoint. Relatively new distribution channels, including the Internet for movies and books, and mobile phones for music, can be embedded with software that provides recommendations in the distribution process. These channels also generate detailed data on customer behavior and preferences. Of course, while these channels can provide a great deal of information about products, there is usually not enough bandwidth or time available for consumers to make truly effective choices. And the smaller the aperture to the customer (a mobile phone screen, for example), the more important it is to assist the customer in making choices, because the amount of information able to be displayed at one time is so limited.

The best reason to use recommendations, however, is that they seem to work — at least for consumers (predictions of success for creators are too new to judge effectiveness). Netflix, for example, has found that customers like its recommendations about 10% better (half a star in their five-star rating system) than their own selections. O2 PLC, a U.K.-based mobile-network operator, found that 97% of its customers opted to use a service to predict and present mobile content that matches their tastes. The Hollywood Stock Exchange aggregates virtual bets from several hundred thousand players on which movies, stars and directors will prosper or decline. A high total of bets in a simulated currency indicates a prediction of success, and few or low bets a failure. One study found that the Exchange’s prerelease predictions of a movie’s box-office take were quite accurate, and comparable to the best expert predictions.16

(Reprint #:50207)

Pages: 1 2