Is there a true public opinion

Some public opinion surveys conducted by media organizations have been criticized for its inprecision. The large gap between poll results and the real election results in the 2004 presidential election has illustrated the problem vividly.

A commentary published in BBC Chinese.com pointed out that "one" medium close to the pan-blue camp has long suffered from the high refusal rate, thereby decreasing the reliability of its polls. This commentary was taken as an evidence for many pan-green supporters to attack the KMT and the media friendly to the party.

However, if you read the article, the sentence criticizing the polls conducted by a certain medium was nothing more than an "observation" or "anecdote." No systematic analysis or research is involved. This article was cited by many bloggers just because it came from the internationally acclaimed BBC (British Broadcasting Company).

Here, I am not going to criticize or defend the pollsters. What I care about is what affects survey responses.

In many Journalism research method class, students have been taught to pay attention to sampling methods and sampling errors. But there are way more than issues related to sampling that would affect survey results.

For example, question wording matters. The effect of wordings on responses was documented in the study of Kahnamen and Tversky about framing. The famous Asian disease experiment can be found here.

Question order is also a factor. If a question about president Chen's job performance is preceded by a question asking about current status of economy, chances are that his rating will be low. This is an effect called "priming," with Iyengar and Kinder (1987) being the early scholars in this field. The effect rests on the assumption that human knowledge is composed of a network of inter-connected nodes. What opinion people may have depends on which "note" is activated.

Other factors, such as the numbering of response categories, the choices provided, question context, and so on, also rendered effects on poll results.

As a media audience, we therefore can not just look at the percentages presented in an opinion poll, for these numbers can be misleading. To be aware of this is very important, especially in an era when people frequently encounter opinion polls and when media coverage focuses only on "who is ahead and who is trailing."