There remains a great deal of debate in the public opinion research industry about the reliability of so-called online research. Online research is typically when a panel is recruited by a company or group of companies to regularly participate in opinion or market research. Each panelist volunteers to participate in research studies conducted by the company or companies. Then, a sampling of the people who have opted-in to the online research panel is selected to participate in any one of these surveys. I don't believe the industry should express paralyzing fear over this kind of methodology and at the same time I join many others who have significant concern about the methodology and how it is promoted.
There are two main reasons one should be skeptical about online panels. The first is when the researcher promotes the results as a representative sample of the broader audience and includes a margin of error. The American Association of Public Opinion Research has some very interesting thoughts on this subject. In fact, the first sentence of the organization's narrative on online surveys reads "The reporting of a margin of sampling error associated with an opt-in or self-identified sample (that is, in a survey or poll where respondents are self-selecting) is misleading" (http://www.aapor.org). Like AAPOR, I believe reporting the results of an online survey as if they are somehow reliant on a normal random sample of a (target) population is misleading. These online surveys are sampling a database of people who have volunteered to participate in these kinds of surveys and not a random sampling of the population whose opinions you seek (i.e. voters). If the survey is not a random sample survey then one can not, and should not apply random probability and calculate a margin of error. Most of these online panel surveys are not based on a scientific random sampling but they are promoted as if they are, which is troubling. At best such surveys will suffer from unknown bias and could possibly skew towards what some have said is a younger, more politically active audience unrepresentative of the broader population. Particularly as a stand-alone methodology, I tend to agree with noted pollster and regular contributor to www.pollster.com , Mark Blumenthal who said "I'd just be, I'd be more cautious about online surveys." Secondly, online surveys are notorious for producing wild results inconsistent with traditional survey methodologies. Again, I will quote AAPOR's website to conclude my concern with these kinds of surveys.
"Surveys based on self-selected volunteers do not have that sort of known relationship to the target population and are subject to unknown, non-measurable biases. Even if opt-in surveys are based on probability samples drawn from very large pools of volunteers, their results still suffer from unknown biases stemming from the fact that the pool has no knowable relationships with the full target population.
"AAPOR considers it harmful to include statements about the theoretical calculation of sampling error in descriptions of such studies, especially when those statements mislead the reader into thinking that the survey is based on a probability sample of the full target population. The harm comes from the inferences that the margin of sampling error estimates can be interpreted like those of probability sample surveys."
Now let me explain why I don't believe the industry should be incredibly fearful of this type of methodology, generally speaking. Technology is creating incredible accessibility to vast numbers of people which can only improve the way opinion and issues research is conducted. In other words, the industry is evolving and the eventual outcomes should be embraced after the reliability and efficacy of such advancements is determined. There will be a time when Blackberries, iPhones and other personal communications devices will have features and applications that allow for research of the user's opinions. We need to maximize the best way to use these technologies to serve our clients and the public while not sacrificing reliability or misleading the end consumers (clients, news media, general public). At MBE we are working on some exciting technologies that can enhance how we conduct our research, and anyone in the industry worth their salt should be doing the same thing. Online surveys are not completely "bad." For instance, I believe as a census survey of a particular audience they can be fairly efficient and cost effective. For example, if a membership organization has a great many members, the organization has access to their email addresses and the ability to design a questionnaire and analyze the results, then an online survey can be appropriate. Especially when the results are not promoted as a random sampling of some broader audience complete with a margin of error. For example the membership organization could conclude that X% of its membership answered a certain question a certain way and that would be appropriate. Also, I believe statistical analysis always calls for a degree of practical thought. The more completed interviews the membership organization received the more reliable the aggregate answers would be (i.e. 400 respondents is more reliable than 40). Making strategic membership decisions based on those 400 respondents would be appropriate.
In the end, online surveys promoted as a scientific random sample surveys are probably not a good idea.
Showing posts with label Mark Blankenship. Show all posts
Showing posts with label Mark Blankenship. Show all posts
Sunday, August 23, 2009
Tuesday, June 16, 2009
Judicial Reform and Public Opinion in West Virginia
In West Virginia, Governor Manchin announced former United States Supreme Court Justice Sandra Day O'Connor will serve as honorary chairwoman of the governor's special commission on the state court system. The commission will examine issues ranging from the partisan election of judges to the creation of a intermediate appellate court. The commission is to provide Governor Manchin with its report in November of this year.
Mark Blankenship Enterprises (MBE) has conducted a number of public opinion surveys in West Virginia on this topic during the last year. Revisiting some of this data helps illuminate how voters perceive some of the very issues which the governor's commission is to examine.
In September of 2008 MBE conducted a random sample survey of 600 registered West Virginia voters (yielding a maximum sample variation of +/- 4% at a 95% confidence level). Sixty percent (60%) disagree electing "supreme court justices by partisan election in which judges are nominated by their political parties such as republican or democrat ... creates a fair court system."
Survey respondents were also read (by live telephone interviewers) a list of three possible ways judges in West Virginia could be selected and asked which they prefer as the best method. The possibilities included "partisan election of judges in which judges are nominated by their political parties such as republican or democrat" which 19% of voters preferred; "nonpartisan election of judges in which the candidate’s political party is unknown" preferred by 36% of voters; and "merit selection of judges in which the governor appoints and legislature approves judges based on the judge’s experience and qualifications" preferred by 37%. Another 8% didn't know which method they prefer.
Two-thirds (73%) of West Virginia voters prefer a judicial selection method other than the current partisan election method. This certainly suggests there is public support for examination and reform such as the commission put forth by the governor. Because a majority or strong plurality doesn't exist for either of the other two questioned selection methods, deeper research is needed to better understand the voter's perceived benefits and concerns with merit selection or non-partisan election. (Methodology Note: Interviewers used random-digit dialing procedure to interview respondents. The numbers are generated by computer to achieve maximum representation in all West Virginia counties. This technique is designed to produce a sample of registered voters that is representative of the entire population in such areas as age, gender, race, and family income. Both listed and unlisted telephone households had an equal chance of being selected in the sample).
It is also important to note survey after survey conducted by our firm in West Virginia indicates the issue which evokes the greatest concern among West Virginia voters is the economy and/or job creation. Moreover, voters regularly express a perceived connection, to some extent, between the the judicial and economic systems or climates of the state. Because the economy is such an intensely concerning issue to voters and because of this perceived correlation between the economy and the judicial system, voters are likely to watch the commission's work with great interest and high expectations. For more information please feel free to visit http://www.markblankenship.com/ or email mark@markblankenship.com.
Mark Blankenship Enterprises (MBE) has conducted a number of public opinion surveys in West Virginia on this topic during the last year. Revisiting some of this data helps illuminate how voters perceive some of the very issues which the governor's commission is to examine.
In September of 2008 MBE conducted a random sample survey of 600 registered West Virginia voters (yielding a maximum sample variation of +/- 4% at a 95% confidence level). Sixty percent (60%) disagree electing "supreme court justices by partisan election in which judges are nominated by their political parties such as republican or democrat ... creates a fair court system."
Survey respondents were also read (by live telephone interviewers) a list of three possible ways judges in West Virginia could be selected and asked which they prefer as the best method. The possibilities included "partisan election of judges in which judges are nominated by their political parties such as republican or democrat" which 19% of voters preferred; "nonpartisan election of judges in which the candidate’s political party is unknown" preferred by 36% of voters; and "merit selection of judges in which the governor appoints and legislature approves judges based on the judge’s experience and qualifications" preferred by 37%. Another 8% didn't know which method they prefer.
Two-thirds (73%) of West Virginia voters prefer a judicial selection method other than the current partisan election method. This certainly suggests there is public support for examination and reform such as the commission put forth by the governor. Because a majority or strong plurality doesn't exist for either of the other two questioned selection methods, deeper research is needed to better understand the voter's perceived benefits and concerns with merit selection or non-partisan election. (Methodology Note: Interviewers used random-digit dialing procedure to interview respondents. The numbers are generated by computer to achieve maximum representation in all West Virginia counties. This technique is designed to produce a sample of registered voters that is representative of the entire population in such areas as age, gender, race, and family income. Both listed and unlisted telephone households had an equal chance of being selected in the sample).
It is also important to note survey after survey conducted by our firm in West Virginia indicates the issue which evokes the greatest concern among West Virginia voters is the economy and/or job creation. Moreover, voters regularly express a perceived connection, to some extent, between the the judicial and economic systems or climates of the state. Because the economy is such an intensely concerning issue to voters and because of this perceived correlation between the economy and the judicial system, voters are likely to watch the commission's work with great interest and high expectations. For more information please feel free to visit http://www.markblankenship.com/ or email mark@markblankenship.com.
Friday, June 5, 2009
Welcome
Thank you for visiting the Public Opinion Research blog published by Mark Blankenship Enterprises, LLC. This blog will cover various research methodologies used to measure and analyze public opinion. We will discuss everything from traditional telephone surveys or polling to focus groups to online web surveys to jury and litigation research and much more. The primary objective of this blog will be to help you, the reader, better understand public opinion research generally, and which quantitative or qualitative research methodologies will best serve your communications needs. We will have a series of guest and expert writers who will contribute to this blog as well. From time to time we will also discuss current events and how such influence and shape public opinion and discuss actual research conducted by Mark Blankenship Enterprises. Relying on our more than 20 years of experience and expertise researching public opinion we hope to make this blog informative, educational and topical.
If you are an attorney seeking to refine and develop case strategy, we will provide insights into the most effective and appropriate litigation research methodologies. If you work for a business or corporation seeking to better understand your customers, the communities in which you operate or improve your image equity, we will discuss community attitude and issues research. If you are an issues advocacy organization or political candidate, we will discuss voter opinion research. If you are a marketing or communications professional, we will discuss effective advertising and consumer research. This blog will provide something for everyone, so sit back and enjoy and check in regularly. Also, feel free to visit http://www.markblankenship.com/.
If you are an attorney seeking to refine and develop case strategy, we will provide insights into the most effective and appropriate litigation research methodologies. If you work for a business or corporation seeking to better understand your customers, the communities in which you operate or improve your image equity, we will discuss community attitude and issues research. If you are an issues advocacy organization or political candidate, we will discuss voter opinion research. If you are a marketing or communications professional, we will discuss effective advertising and consumer research. This blog will provide something for everyone, so sit back and enjoy and check in regularly. Also, feel free to visit http://www.markblankenship.com/.
Subscribe to:
Posts (Atom)
