The Anti-Predictor

From Scientific American:

Early in his new book,  Everything Is Obvious: *Once You Know the Answer (Crown Business, 2011), Duncan Watts tells a story about the late sociologist Paul Lazarsfeld, who once described an intriguing research result: Soldiers from a rural background were happier during World War II than their urban comrades. Lazarsfeld imagined that on reflection people would find the result so self-evident that it didn't merit an elaborate study, because everyone knew that rural men were more used to grueling labor and harsh living standards. But there was a twist, the study he described showed the opposite pattern; it was urban conscripts who had adjusted better to wartime conditions. The rural effect was a pedagogical hoax designed to expose our uncanny ability to make up retrospective explanations for what we already believed to be true. Though Lazarsfeld was writing 60 years ago, 20/20 hindsight is still very much with us. Contemporary psychologists call this tendency to view the past as more predictable than it actually was "the hindsight bias." Watts, a Yahoo! Labs scientist best known for his research on social networks and his earlier book,  Six Degrees: The Science of a Connected Age (W. W. Norton, 2003), argues that this tendency is a greatly underappreciated problem, one that not only causes us to make up just-so stories to explain any conceivable outcome—but to delude ourselves that we can predict the future by learning from the past. (Just because we can create a plausible account of why a book became a bestseller doesn't mean we can tell which new book will be a hit.)

Predictability is elusive because randomness holds much more sway than most of us would like to believe. Drawing on his own research, Watts shows that messages on Twitter don't spread through a predictable set of influential hubs. Similarly, when you ask large numbers of people to relay an e-mail to a stranger through someone they know, there turn out to be no star intermediaries through whom most e-mails find their way. "When we hear about a large forest fire, we don't think that there must have been anything special about the spark that started it," Watts wrote. "Yet when we see something special happen in the social world, we are instantly drawn to the idea that whoever started it must have been special also."

More here.