******************************************************************************************
This article was written in 1990 for the Journal of Small Business Management but was not accepted for publication
SOME PERILS OF MAIL SURVEYS: COMMENT ON FORSGREN
J.J. Ray
University of New South Wales, Australia
Abstract
Forsgren provided readers with techniques for improving the response-rate to mail surveys without giving a warning that use of such techniques may destroy the meaningfulness of the data gathered.
A paper by Forsgren in The Journal of Small Business Management offered a well-informed and useful guide to the carrying out of mail surveys. The paper had a particular focus on how to maximize the response rate in such surveys. People who do surveys do normally want to maximize the response-rate to their surveys in order, presumably, to improve the quality of their data. There seems to be an implicit assumption that increased quantity means increased quality. Forsgren omitted to warn his readers that this is far from so. People who use any of Forsgren's response-rate enhancements may in fact be doing themselves a considerable disservice.
The first intimation I had in my research career that quantity and quality do not go hand in hand occurred when I carried out an attitude survey using Army conscripts as respondents. Since the Army compelled the conscripts to participate I did of course have a 100% response rate. An ideal situation, one might think. Hardly. All that compulsion did was to make non-compliance more subtle. The number of questionnaires with zig-zag response patterns, all answers given as "Not sure" etc was considerable. I could exclude from consideration such obvious attempts to subvert the task but how could I exclude responses such as saying the opposite of what you really think on every second question? The low internal consistencies of the answers given indicated that such things had gone on. I had sacrificed data quality for data quantity.
I suspected that much the same was true in postal surveys and that the more we "squeezed" people to participate the poorer quality data we would get. For this reason, in the many mail surveys I have done I have normally used NO response-rate enhancement techniques. Fairly recently I decided to see if I had been missing anything. Together with a colleague, I sent out a questionnaire that was very similar to many others I had sent out but I used two of the commonest response-rate enhancement techniques on it: I sent out a preliminary letter and I sent out a follow-up questionnaire giving people a second chance to participate. As a result I did get a dramatic boost in response rate. Happy days? Unfortunately, no. The questionnaires I use are fairly sophisticated ones that enable all sorts of internal checks on the consistency of the answers being given and what those checks told us was not good. The data-body we had gathered was heavily infected with meaningless acquiescence (saying "Yes" to almost anything). I had increased my response rate at the price of getting data that was nearly worthless.
The lesson? People just do not respond well to pressure. Even the slight pressures I had used were too much. We rely on voluntary co-operation in survey research and we had better stick with it wholeheartedly if we want meaningful answers. In short, competent as Forsgren's paper was, it may out of sheer self-interest in many cases be best to ignore most of what he says.
Footnotes:
1. Forsgren, R.A. "Increasing mail survey response rates: Methods for small business researchers" Journal of Small Business Management 1989, 27, 61-66.
2. See Study I of Chapter 43 in Ray, J.J. (1974) Conservatism as heresy Sydney: A.N.Z. Book Co.)
3. For details see Ray, J.J. & Still, L.V. (1987) Maximizing the response rate in surveys may be a mistake. Personality & Individual Differences 8, 571-573.
Go to Index page for this site
Go to John Ray's "Tongue Tied" blog (Backup here)
Go to John Ray's "Dissecting Leftism" blog (Backup here)
Go to John Ray's "Australian Politics" blog (Backup here)
Go to John Ray's "Gun Watch" blog (Backup here)
Go to John Ray's "Education Watch" blog (Backup here)
Go to John Ray's "Socialized Medicine" blog (Backup here)
Go to John Ray's "Political Correctness Watch" blog (Backup here)
Go to John Ray's "Greenie Watch" blog (Backup here)
Go to John Ray's "Food & Health Skeptic" blog (Backup here)
Go to John Ray's "Leftists as Elitists" blog (Not now regularly updated)
Go to John Ray's "Marx & Engels in their own words" blog (Not now regularly updated. Backup here)
Go to John Ray's "A scripture blog" (Not now regularly updated)
Go to John Ray's recipe blog (Not now regularly updated -- Backup here)
Go to John Ray's Main academic menu
Go to Menu of recent writings
Go to John Ray's basic home page
Go to John Ray's pictorial Home Page (Backup here)
Go to Selected pictures from John Ray's blogs (Backup here)
Go to Another picture page (Best with broadband)