Correcting the math of journalists in Ohio

Robert Wright in The Atlantic:

ScreenHunter_17 Nov. 06 15.05“The final Dispatch poll shows Obama leading 50 percent to 48 percent in the Buckeye State. However, that 2-point edge is within the survey's margin of sampling error, plus or minus 2.2 percentage points.”

That wording suggests that Obama's two-point edge has no meaning. And that's a common way for journalists to interpret results that fall within the “margin of error.” For example, in September a conservative columnist in the New York Post asserted that Obama's lead in state polls didn't matter because the “polls separating the two candidates are within the margin of error — meaning that there is no statistical difference in support between Obama and Romney.”

Explaining why it's wrong to say there's “no statistical difference” in such cases will take a couple of paragraphs, so please bear with me (unless you recently took a stats course, in which case you can skip these paragraphs if not more).

Here's what pollsters never tell you, except maybe in the fine print: When they say there's a margin of error of X, they don't mean there's no chance whatsoever that the poll is off by more than X. Typically, their margin-of-error calculations are based on a confidence level of 95 percent, which means the chances of the poll being off by more than X are 5 percent.

In the Columbus Dispatch poll, X was 2.2. So if the poll had found Obama was ahead by 2.21 points, the finding would have been “outside the margin of error” and thus treated with great respect — but the fact is that there would still have been a 5 percent chance that, in the actual voting population, Romney was ahead. (I'm assuming the Dispatch followed convention and used the 95 percent confidence threshold in calculating its margin of error — see postscript below.) The flip side of this coin is that the Obama lead of 2 points in the poll, though less than the margin of error of 2.2 points, is by no means devoid of significance. If you did the math — which, many years after taking quantitative analysis in college, I lack the brain cells to do precisely — you'd find that there's a probability of somewhere in the neighborhood of 90 percent that Obama is ahead in the voting population as a whole.

More here.