enquiries@rodlaird.co.uk »
Freephone 0800 612 0910
Freefax 0800 612 0920
Now taking bookings
Bookmark and Share

Why numbers can lie - and how to know when they reveal the truth (01 Jun 19)

What to watch out for in self completion questionnaires


What to watch out for in self completion questionnaires

A magazine wants to find out how its readership really feels about it.

It includes a questionnaire in the back of one of its issues. Answers to some of its key questions suggest that 54 percent think it is important to regularly receive the magazine and 90 percent say their impression of the magazine is highly favourable.

But a high quality sample survey suggests the answers to these two questions are 27 and 79 percent! [i] Why the discrepancies?

The magazine didn’t consider its response rate. It achieved a response rate of only 0.6 percent compared with the high quality sample survey with a response rate 66 percent.

Two useful insights here are that people who don’t respond to questionnaires always have different behaviours and attitudes to those who do and it is always better to spend your money on a small well-conceived and executed sample survey than a poor census.

Overall, a successful self-completion survey requires four elements: (1) a good sample design, (2) a good response rate, (3) a “respondent friendly” questionnaire, and (4) careful planning to stay within budget and maintain quality.

Response rates can be improved through an integrated package of methods all of which stem from social exchange theory. Some examples are multiple reminders with very carefully written letters for each, personalisation of the package, use of token incentives, and early exploration of the reasons for non-response.

A good quality sample design gives every person a chance of selection. And actual selection is made with a true random element rather than subjective choice so that the representativeness of the sample can be backed up with mathematics. This is then coupled with the methods to achieve a good response rate, which focus on obtaining information from all respondents, not just the very co-operative and available.

A questionnaire becomes  “respondent friendly”  through appropriate visual layout. Why is this important? “Many respondents do not read the entire content of questionnaires in a thoughtful way. Respondents take clues from the layout about what must be read and what can safely be ignored, and some respondents skip many words, with the frequent result that questions get misinterpreted. “A well-designed layout prevents items or answer categories from being missed because of their location on the page” i (p. 81).

This is a very rapidly growing field of research. Take the following example.

In the first example above, the conceptual middle of the scale and the visual middle are the same. In the second, the visual middle shifts to “Too little”. Interestingly, respondents’ answers also shift with the visual middle rather than staying anchored around the conceptual middle! [ii]

A good self-completion survey also requires good attention to detail, with careful estimates of both researcher versus other personnel time and other costs such as internal versus external printing, stationery, postage, and indirect costs.

Postal surveys also require careful thinking about how the material should be placed in the envelope (it must all come out of the envelope at the same time with identification numbers matching up – unless one is using a special non-identification method). There also needs to be daily monitoring of returns in a database with mail-merge capabilities.

Working with the four key elements that make a successful self-completion survey will give you a high degree of confidence in your results and their impact on policy decisions.

Dr Pamela Campanelli

Best wishes

 

 

Rod Laird

PS     Looking for more articles on this topic? Check these out.

PPS  Why not run an in house course on sampling? Email Karen at karen@rodlaird.co.uk  

[i]Research discussed by Dillman, D. (2000), Mail and Internet Surveys: The Tailored Design Method, New York, Wiley.

[ii] An example created using the results from Tourangeau, R., Couper, M., and Conrad, F. (2003), Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions, Public Opinion Quarterly, 68(3): 368-393



Why not join the discussion!

Or even better still offer your own advice and tell us about things that others can learn from.

We moderate comments lightly so bear with us and we'll get your thoughts listed as soon as we can.

You must be logged in to post comments.

Not registered yet? Simply fill in the box below.

Email
Password
Forgotten your password?

Like to have access to this and hundreds of other articles like it? Register now!

Just pop your details in the form below, and you'll have full access to our library as well as receiving the free articles you have requested.

Research and evaluation
Resident involvement
Community involvement
Patient involvement
Communications & marketing
Equality and diversity
Community safety
All of these
Your work email

Already subscribed? Want to manage your account? »

Log in

New?

Register now to benefit from hundreds of free hints, tips, articles and interviews

Your email address:

Contact us»

Latest Twitter updates

These courses might interest you:

Advanced statistics

How to conduct a multi channel survey

How to design an effective questionnaire

Why not try these articles?