Real Sampling Error

There have been volumes written about the sampling error in public opinion survey research. In many instances, and especially in political polling, the margin of error is reported by media and others regarding the particular results of a survey. Most any political observer can recall results of a poll which yielded a margin of error plus or minus a certain percentage point. And still sometimes two polls which are sampling the the same audience and subject (i.e. voters during a presidential campaign) may have wildly different results. Both polls may have been conducted by reputable and responsible research firms. On the whole the methodology and instrument (questionnaire) design could have been solid and sound. So how then, can such comparable surveys garner such different results? That question has many possible answers. Professor Cliff Zukin of Rutgers University published an excellent article for the American Association of Public Opinion Research in 2004 titled Sources of Variation in Published Election Polling: A Primer. This article goes on in some detail to discuss all the variables which can cause opinion surveys to have much different results.

Dr. Zukin identifies what is probably the most common culprit and one with which members of the media should become familiar as they report on polling. In a political campaign season we will often hear polls that declare candidate A is ahead of candidate B by a 45 to 40% margin. The survey could have a margin of error of plus or minus 4%. Many times those reporting the survey results will apply the margin of error (plus or minus 4%) to the spread between the two candidates. This would lead one to believe the race could be as close as a 1% lead for candidate A or as high as 9%. That would be true if the margin of error was applied to the spread between the candidates. As Zukin explains, the margin of error is not applied to the spread but rather the percentage points for each candidate. As such, candidate A could have a low of 41% of the vote or as high as 49% . Similarly, candidate B could have a high-water mark of 44% and a potential low of 36%. As such, Candidate A could have range or spread over candidate B as high as 13% (49% to 36%) or Candidate B could have a 3% advantage (44% to 41%). As stated in the article, sampling error (applied correctly) is likely one of the major causes of different results from surveys conducted around the same time and both with acceptable execution.

The entire article can be found at www.aapor.org. Always feel free to visit our website at www.markblankenship.com or follow us on twitter @MCBlankenship