21 December, 2012

IQ tests are 'meaningless and too simplistic' claim researchers

This appears to have been based on an internet survey and such surveys are notorious for giving non-representative results. A large sample size is no substitute for representativeness

The underlying controversy, however, is as old as the hills: Should IQ be measured as a set of subscores or as one overall score? Among psychometricians it is known as the Spearman/Thurstone controvery and dates back to the beginning of the last century.

The accepted answer is to present results both ways: As one overall score plus a set of sub-scores. Results can reasonably be represented both ways because the subscores are correlated. Knowing a person's subscore on (say) verbal ability will give you a useful (but not of course perfect) prediction of his mathematical ability. That has repeatedly been demonstrated.

The novelty in the report below is that the various sub-abilities were said to be NOT correlated -- which runs contrary to 100 years of findings by others. I note however that the authors are more cautious in the underlying journal article. They say:

Using simulations based on neuroimaging data, we show that the higher-order factor “g” is accounted for by cognitive tasks corecruiting multiple networks. Finally, we confirm the independence of these components of intelligence by dissociating them using questionnaire variables. We propose that intelligence is an emergent property of anatomically distinct cognitive systems, each of which has its own capacity.

That sounds to me as if they admit the existence of a general factor but find that the subfactors don't all use exactly the same parts of the brain -- which should be no surprise to anyone.

There is also a question about how comprehensive were the test items used. Without seeing all the questions, I get the impression that a deliberate attempt was made to find questions that would not produce correlated results. One can ask plenty of questions not conceptually related to intelligence and in that case intercorrelations are not be be expected. In psychometrican's terms, the test would lack construct validity.

The journal article is "Fractionating Human Intelligence" by Hampshire et al. I look forward to seeing a more detailed examination of the article by those who specialize in IQ studies

After conducting the largest ever study of intelligence, researchers have found that far from indicating how clever you are, IQ testing is actually rather ‘meaningless’.

In a bid to investigate the value of IQ, scientists asked more than 100,000 participants to complete 12 tests that required planning, reasoning, memory and attention. They also filled in a survey on their background.

They discovered that far from being down to one single factor, what is commonly regarded as intelligence is influenced by three different elements - short-term memory, reasoning, and verbal ability. But being good at one of these factors does not mean you are going to be equally gifted at the other two.

Scientists from Canada’s Western University in Ontario, also scanned some of the participants’ brains while they undertook the tests.

They found that different parts of the brain were activated when they were tested on each of the three factors.

Traditional IQ tests are ‘too simplistic’, according to the research, which found that what makes someone intelligent is too complex to boil down to a single exam.

IQ, which stands for Intelligence Quotient, is an attempt to measure how smart an individual is. The average IQ is 100. Mensa, the high IQ society, only accepts individuals who score more than 148, putting them in the top two per cent of the population.

The new study, published in the journal Neuron, suggests that intelligence is too complex to be represented by a single number.

Study leader Dr Adrian Owen, a British neuroscientists based at Western University in Canada, said an ‘astonishing’ number of people had contributed to the research.

‘We expected a few hundred responses, but thousands and thousands of people took part, including people of all ages, cultures and creeds and from every corner of the world,’ he said.

‘When you take 100,000 people and tested their brain function, we couldn’t find any evidence for a single uniform concept of intelligence.

‘The best we could manage is get it down to three elements that contribute to intelligence. But they are completely different factors, unrelated to one another, and you could be brilliant at one and awful at another. For example, the absent-minded professor.

‘IQ tests are pretty meaningless - if you are not good at them, all it proves is that you are not good at IQ tests.

'It does not say anything about your general intelligence.’ The majority of IQ tests were developed in the 50s and 60s when the way we thought and interacted with the world was different, said Dr Owen.

SOURCE



UPDATE: Chris Brand, a student of IQ, comments on the above study. My comment about sampling appears to have been spot-on:

There was general rejoicing in MSM (e.g. D.Telegraph, 20 xii) as neurocogniwallahs repeated the age-old trick of making the g factor vanish. Using a splendid-sounding ‘sample’ of 44,000 “young and healthy” testees, Adam Hampshire and co-workers reported lots of correlations between mental tests (e.g. verbal, reasoning, memory) that were around a modest .30 (rather than the more usual .50) (Neuron 76, ‘Fractionating human intelligence’). Victory over Burt and Jensen was duly proclaimed.

Any problem with this? Any thought about the astonishing 44,000? Where did they hail from? Ah! “Social networking sites, including Twitter and Facebook.” That is: they were nerds, probably of around IQ 115. And Edinburgh research had demonstrated by 1990 that the g factor ‘fractionates’ above IQ 100 (see TgF, 1996/2000, Chap.2). Yes, Hampshire et al. had well and truly reinvented the wheel – or part of it, for the lower-IQ distribution had been simply forgotten by the ‘neuroscientists.’

No wonder the ‘brain and mind’ ‘natural scientists’ had to publish in an unheard-of journal having no competence to evaluate psychology!

Hopefully Art Jensen and Phil Rushton gave wry smiles from heaven. How one would like to know if Phil had offered his anti-g colleagues at the University of Western Ontario a photocopy of The g Factor!

Called to discuss IQ testing on the Beeb, senior Mensan Peter Bainbridge volunteered that a testee getting only a score around 60 was “probably a carrot” (Daily Mail, 21 xii). An apology was immediately demanded and obtained by irate ‘learning difficulties’ groups – indicating that at least someone still believed in the reality of IQ.



UPDATE 2

There was a further adulatory mention of the Hampshire work over 6 months later -- which I answered rather more succintly here



Go to John Ray's Main academic menu
Go to Menu of longer writings
Go to John Ray's basic home page
Go to John Ray's pictorial Home Page
Go to Selected pictures from John Ray's blogs