The Journal of Social Psychology, 1980, 111, 63-72.
THE COMPARATIVE VALIDITY OF LIKERT, PROJECTIVE, AND FORCED-CHOICE INDICES OF ACHIEVEMENT MOTIVATION*
University of New South Wales, Australia
J. J. RAY
SUMMARY
It was hypothesized that as well as being more reliable than projective tests, behavior inventory measures of achievement motivation would also be more valid. The most valid of the several available such inventories was also sought. Seventy-five Australian males selected by students as "people who you most feel able to rate in terms of their achievement motivation" received a questionnaire containing six behavior inventories, one forced-choice test, and one projective test. Ss rated themselves on achievement motivation after completing the battery and were also rated covertly by those interviewing them. The projective test scores were virtually unrelated to the peer-ratings, while all of the behavior inventories were significantly correlated. The three most valid measures appeared to be the Ray-Lynn scale, the Costello scale I, and the forced-choice Mukherjee "SCT" (roughly in that order). New scales to measure Task orientation and Success orientation as the two most important subcategories of achievement motivation were also developed.
A. INTRODUCTION
The increasing dissatisfaction with projective and, to a lesser extent, forced choice measures of achievement motivation (4, 11, 21) has led many authors to turn to scales in the well-tried Likert format. The present author has collected over 40 such scales. The questions arise, however, "How equivalent are all these measures and if they are not equivalent, which is the most valid?" There is, for a start, ample indication that projective and Likert measures generally fail to correlate at all (6, 9). This is generally attributed to the negligible reliability of the projective measures. Since they are not equivalent, there is, therefore, a clear need to determine what sort of measure offers the highest validity in the measurement of achievement motivation.
B. METHOD
The present study begins with six existing Likert-type scales of achievement motivation and tests them against a peer-rating validity criterion. For the sake of comparison, one projective and one forced choice measure are also included. An attempt is then made to construct from the pool of Likert-type items a new scale which will maximize validity.
The first two Likert scales were the Mehrabian (14) scale and the Ray-Lynn `AO' scale (17, 19). They were selected as representative of global achievement motivation measures. Both had clear prior indications of reliability and validity, plus balance against acquiescent response set. The other four scales included fell into two pairs, each of which reflected an important theory about the nature of achievement motivation. The two McReynolds and Guevara (13) scales were designed to measure the two putatively separate subcategories of achievement motivation called "fear-of-failure" and "success-seeking." The pair by Costello (2) were said to measure "intrinsic" versus "extrinsic" achievement motivation. The projective test chosen was the French (7) "Test of Insight" (FTI); it is the one projective test that emerged from Weinstein's (21) study with some claim to reliability. It was presented at the beginning of the questionnaire. The forced choice test chosen was that by Mukherjee (15) -- the "SCT." It was chosen on account of the extensive validation evidence already available for it (16). Also included in the questionnaire was a short form (8) of the Marlowe-Crowne social desirability scale.
Validity criteria were provided both by the respondent and by the student who contacted him. On the last page of the questionnaire, the respondent gave his occupation and rated himself on achievement motivation. The self-rating question took the form, "How strong would you say your need for achievement was?" The peer-ratings provided by the students carrying out the survey were of eight separate achievement-related attributes: "Task-oriented," "Lackadaisical," "High need for achievement," "Achievement oriented," "Com petitive," "Hard worker," "Leisure-oriented," and "Success-oriented." A single non-motivational peer-rating was also obtained: the degree to which the S was a high "Actual achiever" relative to his opportunities and potentialities. The use of peer-ratings as a validity criterion in contexts such as the present one is a practice that has been argued for by Hollander (10) and Titus (20). The effect of the method is to enable comparison of how the respondent describes himself using the various scales and how others (in this case the person who contacted him) see him.
The battery was administered by a class of second-year undergraduate students to "Males whom you feel able to rate in terms of their achievement motivation." The class was comprised of 13 students in a seminar course on social attitudes and each student took six inventories for administration. Considerable involvement in the project appeared to have been generated and a total of 75 inventories were returned actually completed. Students were also instructed to try to cover as great a range of achievement motivation, from very low to very high, as possible. This entailed a considerable spread in actual achievement and the sample obtained did range from people in the lowest status manual occupations to people in occupations of the highest status. There was however a predictable bias for this type of study to occupations characteristic of the upper to middle classes.
Administration of the inventory was anonymous in that no names were requested on it and the student took it from the respondent and immediately sealed it in an unmarked envelope. The students did however complete separate ratings of the respondent unknown to him and either attached these to the envelope or pre-inserted them in the envelope.
C. RESULTS
The correlations between the various scales and criteria are given in Table 1. It will be seen that the projective FTI is almost totally lacking in validity. Its one significant correlation (with occupational status) is in the opposite direction to that expected [the Congalton status index (1) assigns a high score to people of low status]. Note that all FTI correlations in the table are given after correction for verbal fluency (division of raw score by no. of words used). Without fluency correction, the FTI correlates -.193 with status and nonsignificantly with all other criteria. It may be seen, then, that the choice of which test is the most valid lies between two Likert scales -- the Costello Scale I (intrinsic achievement motivation) and the Ray-Lynn "AO" scale. Against the single most important criterion -- the peer-rating "Need for achievement," the Ray-Lynn scale appears best and the same scale also correlates highest with the self-rating. Another important criterion, however, is the one given as ORAM which is an attempt to deal with the often neglected issue of the reliability of the criterion. It is simply the sum (with appropriate reversals) of all the eight peer-ratings. As a summative scale, its reliability can be estimated by Cronbach's (3) "alpha" -- and is found, in fact, to be high (.85). ORAM, then, stands for Overall Rated Achievement Motivation and is a highly important validity summary. Against it the Costello and Ray-Lynn scales again show the highest correlations.
The next question is whether any of the existing scales maximize reliability. Could new scales be constructed from the data pool which were more valid than any of the existing ones? The best prospect for a new highly valid scale appeared to lie in a minor revision of the Ray-Lynn scale using the item analyses obtained in both the present study and in Ray (17). The six items showing poorest correlation with the scale total in both item analyses (i.e., Nos. 1, 18, 20, 23, 25, and 26) were deleted and replaced by the first four negative items of the next most valid scale-the Costello I. This resulted in a completely balanced 28-item scale with a reliability of .81 (see appendix). The alterations are sufficiently minor and yet well-founded enough to have little effect on the scale's suitability for general population use. Correlations with validity criteria were as follows: .347 with peer-rated need for achievement (vs. .323 with the unrevised version); .574 with self-rated need for achievement (vs. .583 earlier); .269 with peer-rated actual achievement (.174 earlier); .439 with ORAM (.390 earlier); .311 with occupational status (.330 earlier); and .353 with Manual/non-Manual occupation (.347 earlier). This substantial increase in validity as well as reliability suggests that the Ray-Lynn scale should be used in future only in its revised form.
Next the results of a structural analysis of the entire battery of Likert items can be examined. In spite of its limitations, factor analysis has long had and will continue to have an important and empirically useful role in exploring any data set and thus producing or suggesting hypotheses for later test. The analytical method chosen on the present occasion was McQuitty's (12) "Elementary Factor Analysis," really a simple form of cluster analysis. This method was chosen for the reasons given in Ray (18).
At the first order level there were 20 clusters, all of which were nameable but with varying degrees of certainty. Since most of these clusters, however, were of necessity composed of only a few items each, it is the second order clusters that are of most interest. There were three of these-of 57, 28, and 15 items, respectively. The reliabilities for each were .79, .79, and .70 respectively. The first two were clearly identifiable as reflecting Task (intrinsic) and Success (extrinsic) orientation, respectively. The third seemed to reflect Tolerance of Difficulty. The correlation between the first two clusters was, at .196, barely significant at the .05 level. The third cluster correlated .246 and .352 with the first two clusters. The apparent identity of the first two clusters was amply confirmed by their correlation with other variables. The Task orientation cluster correlated .318 with peer-rated task orientation, while the Success orientation cluster correlated only .042 (nonsignificant) with the same rating. With the success orientation peer rating the picture was not quite as neat but, at .431, the correlation with the success orientation cluster was at least considerably higher than the correlation with the Task orientation cluster (.236). The Task orientation cluster also correlated .715 with the equivalent Costello scale (Scale I) while correlating only .395 with the Costello scale II. The success orientation cluster correlated .421 with the equivalent Costello scale (Scale II) and negatively (-.322) with,the Costello scale I. If it were desired to use these two clusters as scales, then, it could be said of them that they have not only satisfactory reliability but also demonstrated predictive and concurrent validity. Interestingly, rated need for achievement correlated similarly with both clusters (.222 and .252), indicating that it is a composite of the two orientations. This was also true of the correlations with the Ray-Lynn scale (.533 and .617), confirming that this scale does measure that composite known as need for achievement. Much the same was also true of the revised Ray-Lynn scale (rs of .653 and .586). Examination of the correlations between the two clusters and rated actual achievement confirms that they too are affected by both orientations -- as would be expected (rs of .335 and .275). Occupational status, however, is related to Success orientation (.340) and not to Task orientation (-.164). This latter finding again is as might be expected.
While the clusters revealed here do then have interesting properties, they are not immediately suitable for use as scales in their own right. To remedy this, shortened versions (see appendix) of the clusters were produced by selecting from each those items which correlated most highly with the total cluster. Thus two new scales of 10 and 16 items each were produced. Their slightly altered validity properties can be noted in Table 1. Their reliabilities were, respectively, .78 and .80.
A final finding of some interest in the present set of data was the failure of the two McReynolds and Guevara scales (success-seeking and failure-avoidance) to measure separate things. At .583, the correlation between them appeared to be limited only by their reliabilities ( 51 and .43, respectively).
D. DISCUSSION
That two almost orthogonal higher order factors have been revealed here which reflect an important conceptual distinction, that this finding represents cross-validation of two previous studies (2, 5) and that the two factors have been shown to have discriminant and convergent predictive validity and discriminant and convergent concurrent validity should combine to make it very clear that discussion of the need for achievement as a unitary trait or attribute can be seriously misleading. Need for achievement is revealed as a somewhat artificial combination of two orientations with quite different theoretical and empirical (predictive) properties. This conclusion, however, is scarcely new in this field. The possibility that need for achievement might be broken up into other dispositions has often been canvassed before. The difference is in what break-up the data support. The confirmation in the present study of the findings by McReynolds and Guevara (13) suggests that a break-up into fear of failure versus success-seeking is empirically unsatisfactory. Scales written to measure these two attributes in fact correlate extremely highly (here .583) and the cluster analysis revealed no new clusters identifiable with these concepts.
In spite of the low correlation (.196) between the two clusters reported above, there is still some validity in having an overall need for achievement index. This is shown by the correlation between the refined versions of these clusters. This was .382, which is amply significant. The raw clusters do of course include a much larger proportion of items correlating negligibly with the total score. It is therefore the item-analyzed (refined) versions of the clusters to which we should primarily look in evaluating the attribute they measure.
APPENDICES
A. The Revised Ray-Lynn "AO" Scale
Reliability .81. "Yes" scored 3, "?" scored 2, "No" scored 1. Items marked "R" scored in reverse.
Now we move on to some questions about yourself. They are about the way you personally behave, feel, and act. We are not trying to "get at" you in any way, so please feel free to be as frank as possible. All answers will be strictly confidential. You circle a number to indicate "yes," "?," or "no." Please work quickly and don't spend time over any one question.
1. Is being comfortable more important than getting ahead? (R)
2. Would you rather not have responsibility for other people? (R)
3. Are you satisfied to be no better than most other people at your job? (R)
4. Do you like to make improvements in the way the organization you belong to functions?
5. Does inefficiency make you angry?
6. Do you take trouble to cultivate people who may be useful to you in your career?
7. Do you find it easy to forget about your work outside normal working hours? (R)
8. Can you forgive a colleague being incompetent so long as he is a nice fellow? (R)
9. Do you like gambling on football pools, raffles, the races, etc. ? (R)
10. Do you get restless and annoyed when you feel you are wasting time?
11. Do you feel irritated when your watch does not keep time properly?
12. Do you limit your recreational and social activities in order to work more effectively?
13. Have you always worked hard in order to be among the best in your own line (school, organization, profession, etc.)?
14. Do you think it wise for young men to delay getting married until they are properly settled in their careers?
15. Do you like getting drunk? (R)
16. Do you dislike seeing things wasted (food, electricity, etc. )?
17. Do you think that success in life is largely a matter of luck? (R)
18. Do you generally try to do jobs as thoroughly as possible?
19. Would you prefer to work with a-congenial but incompetent partner rather than with a difficult but highly competent one? (R)
20. Do you pay a great deal of respect to people in positions superior to your own?
21. Do you tend to spend your money without much planning for the future? (R)
22. Do you tend to plan ahead for your job or career?
23. Is "getting on in life" important to you?
24. Are you an ambitious person?
25. Are you inclined to read of the successes of others rather than do the work of making yourself a success? (R)?
26. Would you describe yourself as being lazy? (R)
27. Will days often go by without your having done a thing? (R)
28. Are you inclined to take life as it comes without much planning? (R)
B. The Items of the New Task and Success Orientation Scales
l. Task-Orientation:
1. Would you rather not have the responsibility for other people? (R)
2. Do you like to make improvements in the way the organization you belong to functions?
3. Does inefficiency make you angry?
4. Do you limit your recreational and social activities in order to work more effectively?
5. Have you always worked hard in order to be among the best in your own line (school, organization, or profession etc. )
6. Do you like getting drunk? (R)
7. Would you prefer to work with a congenial but incompetent partner rather than with a difficult but highly competent one? (R)
8. Do you work for success rather than daydream about it?
9. Would you describe yourself as being lazy? (R)
10. Will days often go by without you having done a thing? (R)
11. Do you do things today rather than put them off until tomorrow?
12. Do you work hard at a job?
13. Do you have a tendency to give up easily when you meet difficult problems? (R)
14. Do you prefer to be an observer rather than a participant because one learns more and gets into less trouble? (R)
15. Do you like to avoid responsibilities and obligations? (R)
16. Are you inclined to take life as it comes without much planning? (R)
2. Success Orientation:
1. Are you satisfied to be no better than most at your job? (R)
2. Do you take trouble to cultivate people who may be useful to your career? 3. Do you find it easy to forget about work outside normal working hours? (R)
4. Is "getting on in life" important to you?
5. Do you have very strong desires to be a success in the world?
6. Are you very interested in the lives of successful people?
7. Are you an ambitious person?
8. Do you readily forget about your work when on holiday? (R)
9. Do you prefer a job which is important, difficult, and involves a 50% chance of failure to a job which is less important but not difficult?
10. Do you think more about your past accomplishments than any future goals? (R)
REFERENCES
1. CONGALTON, A. A. Status and Prestige in Australia. Melbourne, Australia: Cheshire, 1969.
2. COSTELLO, C. G. Two scales to measure achievement motivation. J. of Psychol., 1967, 66, 231-235.
3. CRONBACH, L. J. Coefficient alpha and the internal structure of tests. Psychometrika, 1951, 16, 297-334.
4. ENTWISLE, D. R. To dispel fantasies about fantasy-based measures of achievement motivation. Psychol. Bull., 1972, 77, 377-391.
5. FEATHERMAN, D. L. Achievement orientations and socioeconomic career attainments. Amer. Sociolog. Rev., 1972, 37, 131-143.
6. FISCH, R., $t SCHMALT, H. D. Vergleich von TAT und Fragebogendaten der Leistungs motivation. Z. Exper. & Angew. Psychol., 1970, 17, 608-634.
7. FRENCH, E. Development of a measure of complex motivation. In J. W. Atkinson (Ed.), Motives in Fantasy, Action, and Society. Princeton, N.J.: Van Nostrand, 1958.
8. GREENWALD, H. J., & SATOW, Y. A short social desirability scale. Psychol. Rep., 1970, 27, 131-135.
9. HIMELSTEIN, P., ESCHENBACH, A. E., & CARP, A. Interrelationships among three measures of need achievement. J. Consult. Psychol., 1958, 22, 451-452.
10. HOLLANDER, E. P. The reliability of peer nominations under various conditions of administration. J. Appl. Psychol., 1957, 41, 85.
11. KLINGER, E. Fantasy need achievement as a motivational construct. Psychol. Bull., 1966, 66, 291-308.
12. MCQLiITTY, L. C. Elementary factor analysis. Psychol. Rep., 1961, 9, 71-78.
13. MCREYNOLDS, P., & GUEVARA, C. Attitudes of schizophrenics and normals toward success and failure. J. Abn. & Soc. Psychol., 1967, 72, 303-310.
14. MEHRABIAN, A. Male and female scales of the tendency to achieve. Ed. Psychol. Meas., 1968, 28, 493-502.
15. MUKHERJEE, B. N. A forced choice test of achievement motivation. J. Ind. Acad. Appl. Psychol., 1965, 2, 85-92.
16. MUKHERJEE, B. N., & SINHA, R. Achievement values and Self-Ideal discrepancies in college students. Personality, 1970, 1, 275-301.
17. RAY, J.J. (1970) Christianism.... The Protestant ethic among unbelievers. J. Christian Education, 13, 169-176.
18. RAY, J.J. (1973) Factor analysis and attitude scales. Australian & New Zealand Journal of Sociology 9(3), 11-13.
19. RAY, J.J. (1975) A behavior inventory to measure achievement motivation. J. Social Psychology 95, 135-136.
20. TITUS, H. E. F scale validity considered against peer nomination criteria. Psychol. Rec., 1968, 18, 395-403.
21. WEINSTEIN, M. S. Achievement motivation and risk preference. J. Personal. & Soc. Psychol., 1969, 13, 153-172.
School of Sociology, University of New South Wales P.O. Box 1, Kensington, New South Wales, Australia 2033
* Received in the Editorial Office, Provincetown, Massachusetts, on January 8, 1979. Copyright, 1980, by The Journal Press.
POST-PUBLICATION ADDENDUM
Replication is one of the cornerstones of science. A new research result will normally require replication by later researchers before the truth and accuracy of the observation concerned is generally accepted. If a result is to be replicated, however, careful specification of the original research procedure is important.
In questionnaire research it has been my observation that the results are fairly robust as to questionnaire format. It is the content of the question that matters rather than how the question is presented (But see here and here). It is nonetheless obviously desirable for an attempted replication to follow the original procedure as closely as possible so I have given here samples of how I presented my questionnaires in most of the research I did. On all occasions, respondents were asked to circle a number to indicate their response.
Go to Index page for this site
Go to John Ray's "Tongue Tied" blog (Backup here)
Go to John Ray's "Dissecting Leftism" blog (Backup here)
Go to John Ray's "Australian Politics" blog (Backup here)
Go to John Ray's "Gun Watch" blog (Backup here)
Go to John Ray's "Education Watch" blog (Backup here)
Go to John Ray's "Socialized Medicine" blog (Backup here)
Go to John Ray's "Political Correctness Watch" blog (Backup here)
Go to John Ray's "Greenie Watch" blog (Backup here)
Go to John Ray's "Leftists as Elitists" blog (Not now regularly updated)
Go to John Ray's "Marx & Engels in their own words" blog (Not now regularly updated)
Go to John Ray's "A scripture blog" (Not now regularly updated)
Go to John Ray's recipe blog (Not now regularly updated -- Backup here)
Go to John Ray's Main academic menu
Go to Menu of recent writings
Go to John Ray's basic home page
Go to John Ray's pictorial Home Page (Backup here)
Go to Selected pictures from John Ray's blogs (Backup here)
Go to Another picture page (Best with broadband)