20 Questions A Journalist Should Ask About Poll Results - continued
12. Who's on first?
Sampling error raises one of the thorniest problems in the presentation of poll results: For a horse-race poll, when is one candidate really ahead of the other?
Certainly, if the gap between the two candidates is less than the sampling error margin, you should not say that one candidate is ahead of the other. You can say the race is "close," the race is "roughly even," or there is "little difference between the candidates." But it should not be called a "dead heat" unless the candidates are tied with the same percentages. And it certainly is not a "statistical tie" unless both candidates have the same exact percentages.
And just as certainly, when the gap between the two candidates is equal to or more than twice the error margin—6 percentage points in our example—and if there are only two candidates and no undecided voters, you can say with confidence that the poll says Candidate A is clearly leading Candidate B.
When the gap between the two candidates is more than the error margin but less than twice the error margin, you should say that Candidate A "is ahead," "has an advantage" or "holds an edge." The story should mention that there is a small possibility that Candidate B is ahead of Candidate A.
When there are more than two choices or undecided voters—virtually in every poll in the real world—the question gets much more complicated.
While the solution is statistically complex, you can fairly easily evaluate this situation by estimating the error margin. You can do that by taking the sum of the percentages for each of the two candidates in question and multiplying it by the total respondents for the survey (only the likely voters if that is appropriate). This number is now the effective sample size for your judgment. Look up the sampling error in a table of statistics for that reduced sample size, and apply it to the candidate percentages. If they overlap, then you do not know if one is ahead. If they do not, then you can make the judgment that one candidate has a lead.
And bear in mind that when subgroup results are reported—women or blacks or young people—the sampling error margin for those figures is greater than for results based on the sample as a whole. Be very careful about reporting results from extremely small subgroups. Any results based on fewer than 100 respondents are subject to such large sampling errors that it is almost impossible to report the numbers in a meaningful manner.