What makes a good IQ story?

A good story is one that people read. It may not be true, but it has to meet a need. What do people need when it comes to IQ? That it can be boosted? Certainly, it would be good to have a higher IQ, so long as it does not take too much effort. That IQ is not what it is cracked up to be? Most certainly, in that no self-respecting person wants to be tied down to one number, however predictive that may be. That some of the awkward findings about group differences (particularly racial differences) can be shown to be wrong? Great, now you’re talking. What’s the story?

As carried by The Independent, a London newspaper, the story (21 Dec 2012) was that “IQ tests are 'fundamentally flawed' and using them alone to measure intelligence is a 'fallacy', study finds”. Alongside this confident headline was a dramatic X-ray of a human skull, showing the venous distribution into the brain. Case proved.

The idea that intelligence can be measured by IQ tests alone is a fallacy according to the largest single study into human cognition which found that it comprises of at least three distinct mental traits.

IQ tests have been used for decades to assess intelligence but they are fundamentally flawed because they do not take into account the complex nature of the human intellect and its different components, the study found.

The results question the validity of controversial studies of intelligence based on IQ tests which have drawn links between intellectual ability race, gender and social class and led to highly contentious claims that some groups of people are inherently less intelligent that other groups.

Instead of a general measure of intelligence epitomised by the intelligence quotient (IQ), intellectual ability consists of short-term memory, reasoning and verbal agility. Although these interact with one another they are handled by three distinct nerve “circuits” in the brain, the scientists found.

“The results disprove once and for all the idea that a single measure of intelligence, such as IQ, is enough to capture all of the differences in cognitive ability that we see between people,” said Roger Highfield, director of external affairs at the Science Museum in London.

“Instead, several different circuits contribute to intelligence, each with its own unique capacity. A person may well be good in one of these areas, but they are just as likely to be bad in the other two,” said Dr Highfield, a co-author of the study published in the journal Neuron.

The research involved an on-line survey of more than 100,000 people from around the world who were asked to complete 12 mental tests for measuring different aspects of cognitive ability, such as memory, reasoning, attention and planning.

The researchers took a representative sample of 46,000 people and analysed how they performed. They found there were three distinct components to cognitive ability: short-term memory, reasoning and a verbal component.

Professor Adrian Owen of the University of Western Ontario in Canada said that the uptake for the tests was astonishing. The scientists expected a few hundred volunteers to spend the half hour it took to complete the on-line tests, but in the end they got thousands from every corner of the world, Professor Owen said.

The scientists found that no single component, or IQ, could explain all the variations revealed by the tests. The researcher then analysed the brain circuitry of 16 participants with a hospital MRI scanner and found that the three separate components corresponded to three distinct patterns of neural activity in the brain.

“It has always seemed to be odd that we like to call the human brain the most complex known object in the Universe, yet many of us are still prepared to accept that we can measure brain function by doing a few so-called IQ tests,” Dr Highfield said.

“For a century or more many people have thought that we can distinguish between people, or indeed populations, based on the idea of general intelligence which is often talked about in terms of a single number: IQ. We have shown here that’s just wrong,” he said.

Studies over the past 50 years based on IQ tests have suggested that there could be inherent differences in intelligence between racial groups, social classes and between men and women, but these conclusions are undermined by the latest findings, Dr Highfield said.

“We already know that, from a scientific point of view, the notion of race is meaningless. Genetic differences do not map on to traditional measurements of skin colour, hair type, body proportions and skull measurements. Now we have shown that IQ is meaningless too,” Dr Highfield said.

Where to start? It would seem that there are persons walking about this earth, entirely unsupervised and with access to resources, but with any luck not to heavy machinery, who think that you can make statements about human cognition on the basis of 16 subjects. The level of insolent innumeracy makes one’s jaw drop. 16 persons do not humanity make. The sole description of these paragons is that they were healthy and young. No mention of their occupations or ability levels, or anything else about them.

Talking as I do to intelligence researchers involved in brain scanning and intelligence, they willingly concede that brain scans are not immune from the requirements of sampling theory. Simple power calculations suggest that sample sizes of 200 or so would be the minimum required for reliable findings. Lars Penke (University of Edinburgh) recently presented an interesting and sound paper on brain-wide white matter tract integrity and general intelligence with 420 older adults, supporting the Parieto-Frontal Integration Theory model. Richard Haier (University of California at Irvine) and his colleagues aim to go beyond that, and intend to treat intelligence brain scan studies with the same care as the standardisation of conventional psychometric tests.

Against those demanding standards few of the popularly proposed alternative tests would survive. You need sample sizes of about 1,400, you need to show that you have a good representation of the population in question (intelligence range, urban/rural balance, very good age representation) and in addition you need to double sample minorities i.e. if pure sampling theory indicates you should test 200 African Americans you have to test 400, simply to have better confidence limits. After all that, you are still open to legal challenge if any of the items even remotely appear to be biased against any group (determined by any one group doing particularly badly on that item). Only then can your sparkling new test be released to the public, and the process takes several years, and considerable effort and money.

Second, large numbers of self-selected subjects do not get round selection biases: they confirm them. I do not visit sites discussing the likelihood of Elvis Presley being alive. Sites offering an intelligence test tend to attract brighter subjects. Access to websites of any sort requires some IQ, as do voting machines. By means of voluntary computer testing one could easily exclude the bottom 25% of the population. g is extracted from the whole range of intelligence, and is weaker at the higher levels. Chris Brand has covered this point many times. The paper does not discuss this artefact.

Third, assume that a good study, not this one, were to show beyond doubt that intelligence could be decomposed into three factors. Would this do away with group differences? It would only be of interest if one or two of those factors had greater real life predictive value than the overall extracted g factor and that this factor ran against the usual hierarchy revealed by g. In that instance it would be possible to argue that one group was behind on g but ahead on spatial skills, and this was somehow far more important in life.  This is highly unlikely, and flies in the face of a century of psychometric results on group differences, but it would be very interesting if it could be shown that this was the case. It cannot be assumed as an act of faith, as the authors have done.

Most interestingly, the paper as published bears little relation to the interviews the authors gave. The paper per se is concerned with a model of the brain, and a component analysis of a set of computer administered mental tasks. As discussed above, if the model were to be applied to a proper sample, then the results might be interesting. Revealingly, the interviews go well beyond the sparse findings, to grandiose claims about the end of g as a construct, and the end of meaningful group differences. To counterbalance the shortage of scanned subjects there has been a surfeit of boasting.

How did this paper get such adulatory press coverage? It told a story that people wanted to believe. It ticked all the boxes required by wishful thinking. Cold fusion, anybody?

And a happy 2013.