Monday, April 30, 2007

Is Predicting Hit Songs Futile?

I recently covered Columbia Professor Duncan Watts’ “cumulative advantage” experiment, in which similar groups of people started with the same selection of songs but ended up with different choices for which songs were hits. If people were just judging the songs on content, the groups’ choices for hits should have been similar. However, there was also a social factor: Except for a control group, each group’s members could see the popularity of songs within their group but not within other groups.

Professor Watts proposed that the divergent choices for hits were due to each group’s piling-on to whatever happened to be initially popular within that group. See the original post for details.

In my praising the experiment, I held back on some questions about the strongest claim in Professor Watts’ New York Times Magazine article. In essence, he claimed that predicting hits was futile due to the inherent randomness of social systems like the word-of-mouth that affects entertainment choices:

Because the long-run success of a song depends so sensitively on the decisions of a few early-arriving individuals, whose choices are subsequently amplified and eventually locked in by the cumulative-advantage process, and because the particular individuals who play this important role are chosen randomly and may make different decisions from one moment to the next, the resulting unpredictability is inherent to the nature of the market.

This effect was true of Professor Watts’ experiment, but is it realistic to have the early-arriving individuals “chosen randomly”? Isn’t there a relatively small percentage of people who act as tastemakers: people who are into new stuff first and whose knowledgable opinions influence others? If these people have non-random qualities, shouldn’t there be a lot more predictability?

The Limits of an Individual Influential

After some email back-and-forth with Professor Watts, I was surprised to find that the role of “influentials” is potentially a lot less than is commonly believed. In a draft of a paper due for publication later this year, Watts and collaborator Peter Dodds detailed their mathematical simulations of various scenarios involving influentials. The results were summarized in the Harvard Business Review’s Breakthrough Ideas for 2007:

Our work shows that the principal requirement for what we call “global cascades”—the widespread propagation of influence through networks—is the presence not of a few influentials but, rather, of a critical mass of easily influenced people, each of whom adopts, say, a look or a brand after being exposed to a single adopting neighbor. Regardless of how influential an individual is locally, he or she can exert global influence only if this critical mass is available to propagate a chain reaction.

To be fair, we found that in certain circumstances, highly influential people have a significantly greater chance of triggering a critical mass—and hence a global cascade—than ordinary people. Mostly, however, cascade size and frequency depend on the availability and connectedness of easily influenced people, not on the characteristics of the initiators—just as the size of a forest fire often has little to do with the spark that started it and lots to do with the state of the forest.

The researchers’ forthcoming paper makes a compelling case for these conclusions, exploring influentials’ role under many different scenarios. However, its various social-network models all start with the single “spark” of an individual discovering and communicating something. It does not consider a scenario where a large number of simultaneous and non-random sparks occur throughout the network. That is, if a single, random spark can cause a forest fire under the right conditions, how about a bunch of sparks purposely set at once, across that same forest?

The Potential of Coordinated Influence

The coordinated, multi-spark scenario matters because it is how certain social-marketing companies supposedly work: unleashing a small army of “on message” people to tell their friends about some great new thing. One might argue that a favorable newspaper review, radio airplay, or other one-to-many media do something similar, simultaneously “sparking” many consumers at once.

The key point: Instead of having a single line of sentiment that needs to propagate enough times to reach critical mass, the multi-spark scenario has many lines propagating, each of which could randomly run into other lines, thereby accelerating toward a critical mass.

Bringing this all back to the original New York Times Magazine article and its assertion that hits cannot be predicted, the multi-spark scenario is a way for hits to be predicted. In essence, it increases the prediction reliability by manipulating the system.

You may say this is unfair, like loading the dice, but it’s how entertainment marketing works. Companies spend marketing dollars in proportion to what they think will be popular, thereby making what they think will be popular more popular. Economically, the question is whether the cost of manipulating the word-of-mouth system is worth the increased probability of a hit.

Note that predicting a hit doesn’t mean being right all the time; it just means that across many attempts the gain is greater than the cost. Thus, even if you only went from a 3% hit rate to a 5% hit rate, predicting was worthwhile if it cost less than the benefit from those extra two percentage points.

So, I’m not ready to conclude that it’s futile for entertainment companies to predict hits. If the companies were merely acting as pure observers, then Professor Watts’ case would be strong enough for me. However, because entertainment companies’ predictions are often entangled with manipulating the systems being predicted, there may still be reason to try to pick winners.

Whether the benefits outweigh the costs...well, that’s another experiment to do.

8 comments:

  1. Prediction by manipulation! Nice. Wouldn't it be great if it worked in, say, roulette?
    The original question is still unaddressed -- can a hit be predicted without manipulation?
    I read Watts' piece, went to their site, and played the game. The biggest problem there is that the quality of the songs is pretty uniform and uniformly low. No wonder the results are random. I am afraid that the way it was conducted, Watts' experiment proves or disproves nothing.

    ReplyDelete
  2. I haven't seen the forthcoming paper, but I did read the original one in Science, and, based partly on that, I have a couple of reservations about what seems to be Watts' interpretation.
    Firstly, "the principal requirement for what we call global cascades is the presence not of a few influentials but, rather, of a critical mass of easily influenced people, each of whom adopts, say, a look or a brand after being exposed to a single adopting neighbor". That's kind of like saying the principal requirement for a domino effect is not someone to tip the first domino, but the presence of lots of closely arranged dominoes. It's true, but it's not the whole story. You can argue whether one requirement is 'principal' and the other 'secondary', but surely you need both of them?
    The second reservation is about whether some people are better placed to 'tip the first domino' than others. In Watts et al's first study, none of the people whose selections made up the 'charts' had the scope to be more influential than others - they didn't know each other and they couldn't get to know each other. As with many laboratory-style experiments, the workings of Power and History were excluded by design. But in the real world, you can't ignore Power and History. Some people are in more influential positions: they have more dominoes closer to them or they have a better reach when it comes to tipping other dominoes. And, over time, people acquire a reputation, which makes others more susceptible to being influenced by them.
    It's the workings of Power and History that tend to concentrate influence in the hands of a small minority (and also to create a critical mass of easily influenced people).
    As I say, I haven't seen the later paper myself, but I'd be interested to know whether it addresses these factors.

    ReplyDelete
  3. Gene: If the study participants as a whole found the songs as uniform as you did, the ratings and downloads for the independent group would have been uniform. That was not the case.

    ReplyDelete
  4. David: Regarding your domino analogy, Watts et al stress the importance of the system over the first mover, although both are necessary. What's not necessary, they argue, is the need for the first mover to be "special" (like an expert or otherwise an influential).
    As for your second reservation, the forthcoming paper models overlapping groups of acquaintances within a population, whereby people in an acquaintance group have greater influence on each other.

    ReplyDelete
  5. Steve -- I disagree. In a totally random network, supernodes form naturally anyway. The fact that ratings or downloads were not uniform does not prove that popularity is independent from quality. It only proves that in absence of meaningful quality differences, popularity is influenced by the social network, which simply apmlifies initial random preferences of a few. A rather trivial thought, isn't it.

    ReplyDelete
  6. >> It only proves that in absence of meaningful quality differences, popularity is influenced by the social network <<
    Gene: Just to make sure we're talking about the same thing, my previous comment referenced the "independent group": the one with no social influence. In essence, it was the control group for the experiment.
    The fact the *independent group* was not uniform in ratings and downloads indicates: (1) Those in the group perceived meaningful quality differences, and (2) the differences were not due to social influence.

    ReplyDelete
  7. Steve -- I guess we differ in the degree of meaningfulness of the control group's perceptions. It would be interesting to measure the strength of these perceptions by having 10 independent control groups. My bet is that we'd see very different distributions across those groups.

    ReplyDelete
  8. Cumulative advantage is at work in the presidential primaries, courtesy of the networks' obsession with "popularity" - measured in both polls and actual performance at the polls. I've read somewhere that people pile onto the successful candidates.
    Also, didn't see a mention of Malcolm Gladwell's "Tipping Point", but I recall some mostly anecdotal information about early market-makers who influence the trends.

    ReplyDelete