Debate the Usefulness of Election Models

There a debate on the issuse over at the NYT's FiveThirtyEight. First, Nate Silver:

...Lynn Vavreck’s excellent 2009 book, “The Message Matters,” for instance, made the following claim:

The economy is so powerful in determining the results of U.S. presidential elections that political scientists can predict winners and losers with amazing accuracy long before the campaigns start.

To be clear, that is the publisher’s copy and not Ms. Vavreck’s. However, statements like these have become fairly common, especially among a savvy group of bloggers and writers who sit at the intersection of political science and the mainstream media (a space that this blog, of course, occupies).

But is it true? Can political scientists “predict winners and losers with amazing accuracy long before the campaigns start”?

The answer to this question, at least since 1992, has been emphatically not. Some of their forecasts have been better than others, but their track record as a whole is very poor.

And the models that claim to be able to predict elections based solely on the fundamentals — that is, without looking to horse-race factors like polls or approval ratings — have done especially badly. Many of these models claim to explain as much as 90 percent of the variance in election outcomes without looking at a single poll. In practice, they have had almost literally no predictive power, whether looked at individually or averaged together.

John Sides responds:

I am less critical of the accuracy of these models than is Nate. For one, forecasters have different motives in constructing these models. Some are interested in the perfect forecast, a goal that may create incentives to make ad hoc adjustments to the model. Others are more interested in theory testing — that is, seeing how well election results conform to political science theories about the effects of the economy and other “fundamentals.” Models grounded in theory won’t be (or at least shouldn’t be) adjusted ad hoc. If so, then their out-of-sample predictions could prove less accurate, on average, but perfect prediction wasn’t the goal to begin with. I haven’t talked with each forecaster individually, so I do not know what each one’s goals are. I am just suggesting that, for scholars, the agenda is sometimes broader than simple forecasting.

Second, as Nate acknowledges but doesn’t fully explore (at least not in this post), the models vary in their accuracy. The average error in predicting the two-party vote is 4.6 points for Ray Fair’s model, but only 1.72 points for Alan Abramowitz’s model. In other words, some appear better than others — and we should be careful not to condemn the entire enterprise because some models are more inaccurate.

Third, if we look at the models in a different way, they arguably do a good enough job.