Question: PISA and cheating

Has anyone been able to find the part of the technical appendix in PISA 2012 which deals with the issue of whether China cheated? Or, for that matter, whether any other country cheated?

Andreas Schleicher wrote in his blog today that this is a nonsense, and that critics like Time magazine “didn't bother to read the PISA 2012 Technical Background Annex, which shows there was no cheating, whatsoever, involved.”

I can’t find that bit in the technical background annex, and the page to which Schleicher directs readers in his blog is a very general one, including parts which the system says I don’t have access to. I have sent an email requesting access, and have looked through lots of tables but can only find lots and lots of country tables and figures.

Can anyone help direct me to the relevant part of the annex?

ISIR Press Release: Where are the keys to the castle?

 

from the International Society for Intelligence Research Conference in Melbourne Australia. NO EMBARGO.

The story so far is that the DNA of 2,200 extremely intelligent people has been handed over to the Beijing Genetics Institute, so that they can work out the genetics of genius. The preliminary results are to be presented at the Melbourne conference.

The abstracts of the relevant papers are pretty cagey, not to say inscrutable. Deducing whether they have found anything is like working out the power relationships in the Chinese Politburo. The news is that they are still crunching the data on these 2,200 very high ability persons. However, as a warm up they have looked at the top 3% scorers on the cognitive assessments administered to 3 million 18-year-old males as part of compulsory military service in Sweden 1950-1990.

They identified 370,00 sibling pairs and 9,000 twin pairs. The top 3% of intelligence composite scores was familial, heritable, and caused by the same genetic and environmental factors responsible for the normal distribution of intelligence. These results imply that any genetic effects identified in the HiQ project will not be ‘genes for genius’; rather, they will apply to IQ throughout the distribution, including low IQ.

So, not exactly the keys of the inner keep of the castle, but a useful impression of the keys to the outer walls. Genius will probably be just more of what constitutes intelligence for the rest  of us. By the way, they are still looking for more highly intelligent subjects. Tell them I recommended that you apply.

Contact: Prof Robert Plomin robert.plomin@kcl.ac.uk

ISIR Press Release: Like humans, dogs have general intelligence

 

from the International Society for Intelligence Research Conference in Melbourne Australia. NO EMBARGO.

General intelligence in dogs

Arguments have raged as to whether human intelligence contains a large general component, or whether that just arises from the way that IQ tests are constructed. Now Dr Rosalind Arden, Dr Mark Adams and Prof Robert Plomin have tested the intelligence of border collies on four different detour route-finding tasks and two different tests of pointing.

They say: A higher order factor accounted for 40% of the reliable variance in performance. Dogs that quickly completed the detour tasks also tended to score higher on the choice tasks and take less time to make a choice, and this could be explained by a general intelligence factor. The weaker inter-correlations among test performance implies that dog intelligence may have evolved in response to selection for specific behaviors and abilities.

So, if you consider that general intelligence account for 50% of human ability, and 40% of border collie ability they have much in common in terms of mental organisation, but dogs are more likely to have some specific abilities because they were bred up to do specific tasks by humans.

Contact: Dr Rosalind Arden   Rosalind.arden@kcl.ac.uk

If she is on a plane at the moment, I can provide a preliminary summary.

Intelligence in 2000 words

 

I have noticed that in my series of 4 posts “All you ever wanted to know about intelligence (but were too bright to ask)” the first part got far more attention than the others, so I have put them together and boiled them down to 1,985 words.

“Some people are cleverer than others. It is a prominent and consistent way in which people differ from each other; the measurements we make of people’s cleverness produce scores that are correlated with important life outcomes; it is interesting to discover the mechanisms that produce these individual differences; and understanding these mechanisms might help to ameliorate those states in which cognitive function is low or declining.”

Deary identifies four major sources of scepticism about intelligence:

1 The concept appears to be too general. People argue that they are better at some skills than others, and assume different modules are involved, such that we are all good at some specific mental skill.

2 Historical events in intelligence research which are discreditable. In the UK, the 11+ missed out people who later showed demonstrable talent; cases of probable fraud in reporting results; over use and over-interpretation of intelligence tests; controversies about intelligence differences between ethnic groups; or claims that “ordinary” intelligence has now been replaced by tests of “multiple” intelligence.

3 “It is possible that clever people develop a kind of cognitive noblesse oblige; they kind of know they have won the lottery on a valuable trait, but they think it is bad form to acknowledge it.”

4 They probably haven’t read good quality research on the topic.

I find that most of the hostility about intelligence comes from bright people, who keep up with the broad sweep of newspaper reports and popular books, but have not looked at good quality research. Despite this lack they are surprisingly vehement when they ridicule IQ. It would appear that intelligence is most disparaged by the intelligent. “Define intelligence” they demand, with a knowing smile.

There are three types of answer: a quip, an explanation, and a formula.

The Quip: “Intelligence is what you need when you don’t know what to do”. Carl Bereiter coined this elegant phrase. It captures the ultimate purpose of intelligence, which is to help you cope with the unknown. The best intelligence test is the puzzle to which no one knows the answer. For example, is there a detectable particle which gives objects mass?

The Explanation: “Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience. It is not merely book learning, a narrow academic skill, or test-taking smarts. Rather, it reflects a broader and deeper capability for comprehending our surroundings — ‘catching on,’ ‘making sense’ of things, or ‘figuring out’ what to do.” Linda Gottfredson and 52 leading psychometricians agree with this explanation. http://www.udel.edu/educ/gottfredson/reprints/1997mainstream.pdf

The formula: g+group+specific skill+error, where g accounts for about 50% of the variance. (I have written this in English, but it should be displayed in eigenvalues).

So, let us look behind the formula “In 1904 Spearman found that people who perform well on one type of cognitive test tend to perform well on others. That is, if cognitive test scores are ordered so that better performance equals a higher score, the correlations between them are all positive. There is shared variation among all types of cognitive performance. Spearman called this shared/common variance g: an abbreviation for general intelligence. In the 100+ years since then, every study that has applied a diverse battery of cognitive tests to a decent-sized group of people with a mix of ability levels has re-discovered the same thing: there is some cognitive variance shared by all cognitive tests. Typically, if one applies principal components analysis, just under half of the total test score variance is accounted for by the first unrotated principal component.”

This finding of 50% of the variance in ability being due to g is matched by another finding:  IQ-type test scores are highly reliable, and highly stable. For example, when the same intelligence test is taken at age 11 years and repeated at almost 80, about 50% of the variance is stable.

So, half of intelligence is due to a common factor, and half of the variance is stable throughout life.

What are the causes of differences in intelligence? Twin and adoption studies show that intelligence is about 50% heritable. Counter-intuitively, heritability estimates increase to 80% as you age and put many years between yourself and your family’s influence on you. It looks like genes take time to express themselves, or set in motion processes which take time to develop.

If you measure a diverse range of skills you can show that g is highly heritable, but there is less genetic influence that is specific to each domain. Heritabilities are probably higher in richer people, and lower in deprived groups (suggesting the environment has most effect when it is very bad).

There is no “gene for intelligence” with the exception of the APOE gene, where individuals with one or two e4 alleles tend to have lower ability in old age, and declining cognition across their lifetimes.

Genome wide assessment studies haven’t come up with very much, yet. There is some molecular genetic evidence that some variance in intelligence is detected by single nucleotide polymorphisms. Applying genetic complex trait analysis, between a quarter and a half of intelligence variance can be accounted for by variants in linkage disequilibrium with common SNPs. This analysis cannot identify the precise causal genes. It suggests that intelligence is highly polygenic, with large numbers of variants of small effect sizes.

The genetic correlation between intelligence measured in childhood and old age in the same individual is high: to a substantial extent the same genes cause higher intelligence in both childhood and old age. All these studies require very large sample sizes (125,000) and allow major risks to be computed: in individuals who are at genetic risk for schizophrenia, but have not exhibited the disorder, cognitive ability in old age is lower than in those without schizophrenic genetic risk.

With regards to the much vaunted effects of the environment, twin studies suggest that the contribution of the shared environment (family for example) to intelligence is small to negligible by adulthood, and the remaining variance is due to individually created environments and error.

There is a general finding that there is a modest correlation of 0.30 between intelligence test scores and brain size, and a similarly sized correlation between intelligence and the general integrity of the brain’s white matter, as measured by diffusion tensor MRI. Sample sizes here are of the order of 500 persons. The association is largely accounted for by people’s differences in speed of processing. Cleverer brains seem to be more efficient. Consequently, they can think about complex problems faster and for longer, thus being more likely to find solutions.

Education

”People who score better on intelligence tests tend to stay longer in education, to gain higher-level qualifications, and to perform better on assessments of academic achievement. Some of the correlations between intelligence scores at the end of primary school and academic results some years later are high, suggesting that it is not just a matter of education boosting intelligence. Also, educational attainment has a moderately high heritability, and a strong genetic correlation with intelligence. On the other hand, there is also evidence that education can provide a boost to scores on tests of complex thinking, and some of these increments last into
old age. Therefore, there is probably a bidirectional causal association between intelligence and education.”

Social status and mobility

“People who score better on intelligence tests tend to go into more professional occupations (typically those with higher status) and to perform better
in the workplace. There is a positive association between intelligence test
scores in childhood and social position later in life: people who score higher
tend to be in more professional jobs, to live in less deprived areas, and to
have higher incomes.” This is not due to people stating their intelligence test results in their job applications nor, usually, to employers conducting their own intelligence tests (though that would often be useful). These results are obtained by looking at historical intelligence test results years after the children have grown up and finished their professional careers. 

“The association is not perfect. Results show that, when it comes to attained social position in maturity, intelligence, education and parental background all count to
some extent. That is, there is some meritocracy and intelligence-driven social mobility, and there is also some social inertia.”

Intelligence seems to have a cumulative effect, and relates more strongly to occupational and social position later rather than earlier in adulthood.

Intelligence and health

Intelligence is associated with better health and longer lifespans, but it is not entirely clear why. The early explanation was that more intelligent people learned quickly how to avoid health hazards. They gave up smoking sooner in life, bothered to read the medicine labels, and followed health advice generally. Now it seems possible that both intelligence tests and life itself test a general underlying bodily system integrity, a fundamental mens sana in corpore sana which, if you are lucky enough to have it, gives you health, intelligence and long life without much exertion on your part. Typical, isn’t it, that evolution doesn’t understand human concepts of fairness and equity? Also typical that many very good papers were written showing how intelligent people avoided health hazards, and now it turns out that those will have to be re-written.

Age related cognitive decline

“There are declines in cognitive function even among people who do not develop dementia. Not all cognitive functions decline at the same rate. Some cognitive functions — often referred to as markers of crystallized intelligence — hold up well with age. These include vocabulary and general and specific knowledge. The cognitive functions that tend to decline are called fluid intelligence. These tend to involve on-the-spot thinking with novel materials, and in situations in which past knowledge is of limited help. This includes abstract reasoning, spatial abilities, processing speed, and working and other types of memory.

Not everyone experiences the same rate of cognitive decline, and there is a growing interest in the genetic and environmental (biological and social) determinants of people’s differences in age-related cognitive changes. Some of the more solid evidence exists for the following being cognitively protective: not having the APOE e4 allele, being physically more active and fit, and not smoking.”

Two main hypotheses are: some people have a “cognitive reserve” such that their brains are better able to withstand damage, perhaps because a bigger brain provides redundancy or because some people’s brains are more flexible in reorganizing networks to regain or retain cognitive functions; or the common system hypothesis that age-related decline of different bodily systems is correlated; that people who are experiencing faster cognitive declines might also be experiencing faster declines in sensory and some physical functions, making researchers consider inflammation, oxidative stress, telomere length, and the hypothalamic-pituitary-adrenal axis as common causes of variance.

Finally, Deary goes on to discuss the “so-named Flynn effect, whereby the absolute scores on intelligence tests have been rising since testing started in the early-to-mid 20th century. The extent of the rise, its geographical distribution in the world, and especially its causes are all still being studied. Some hypothesise that better nutrition might explain some of the increase, and others put it down to society’s making more accessible and emphasizing the skills tested by intelligence tests.”

“On the biological side there is research showing that breast feeding is associated with a sizeable advantage in intelligence later in childhood. However, there is also some evidence that this is explained by the higher intelligence scores of the mothers who tend to breastfeed.

“Adoption from a deprived to a more affluent setting is reported to be associated
with an intelligence advantage. There is still debate about the effectiveness
of intensive intervention programmes early in life, and whether any cognitive
advantages last or whether advantage accrues to social rather than cognitive
skills.”

Deary ends with a plea: “Human intelligence is important; it matters in our lives.”

Here are his references:

Deary, I.J. (2012). Intelligence. Ann. Rev. Psychol. 63, 453–482.
Deary, I.J., Penke, L., and Johnson, W. (2010). The neuroscience of human intelligence differences.Nat. Rev. Neurosci. 11, 201–211 .
Hunt, E. (2011 ). Human Intelligence (Cambridge: Cambridge University Press).
Nisbett, R. E., Aronson, J., Blair, C., Dickens, W., Flynn, J., Halpern, D. F., and Turkheimer, E. (2012). Intelligence: new findings and theoretical developments. Am. Psychol. 67, 503–504.
Salthouse, T.A. (2010). Major Issues in Cognitive Ageing (Oxford: Oxford University Press).

The wages of Mandela

 

The Economist has produced a graph which shows incomes in South Africa by racial group from 1918 to the present day, which is the lifespan of Nelson Mandela. These are their blunt conclusions:

Income growth improved substantially for all South Africans after Mandela’s 1994 election victory, but sufficiently more so for whites, and the balance has been disproportionately weighted in their favour—and increasingly that of Asian South Africans—since he stepped down in 1999. Under its own majority rule, the lot of the ever-growing black population—today forming over three-quarters of the national total—has been notably poor. Misguided governance, low-quality education, skills shortages and massive unemployment levels of around 40% have left it more disadvantaged today than when Nelson Mandela was still behind bars. Black income has virtually flat-lined, betraying tremendous gulfs between the wealth of the different racial groups. Sadly, the nation Mandela leaves behind today remains one of the most unequal in the world.

If you look at their graph, all they say is true, and they have correctly described what they have drawn. However, they strongly imply that with better governance and better education things would be far better, and Mandela must shoulder part of the blame. I am not opposed to criticism of politicians, but before judging this particular one I have a few questions.  Why are there skill shortages? Why are there so many unemployed citizens? What are black wages like in the rest of Africa?

 

Three different estimates put South Africa on a GDP per capita level of about $4,000 which is equivalent to Botswana, which is generally judged one of the best governed in sub-Saharan Africa, with high levels of honesty and transparency for doing business (about 4th in the world at last count). At a higher level of GDP are Seychelles and Mauritius, but South Africa is near the top of the lists on all GDP estimates. Frankly, if South Africa is close to the top, in the company of Botswana, then it is hard to point the finger at South African black governments for black income levels.

What other differences can we find between whites, asians, mixed race and black citizens in South Africa? To put it mildly, over a century the first three groups have had some babies, but the last group has had many babies.  The black population has risen from about 3 million to about 40 million. That creates opportunities, no doubt, but it creates a lot of competition for jobs, unless the economy is exceptionally well managed in the global market place. (At the same latitude further to the West, over the same time period Uruguay grew from 1.4 to 3.4 million, GDP per capita is $15,800).

Perhaps I have misjudged The Economist. Perhaps they had a conference and after a big row came up with a compromise between the graphics and the editorial department. Each would tell their story, and the readers could work things out for themselves. After all, a separation of wages according to skill levels is entirely consistent with an open, flourishing economy in which everyone seeks opportunity, to the best of their ability.

What else might The Economist have mentioned?

S.Africa -whites	15	1,056	SPM	94	Owen, 1992
S.Africa-blacks	10/12	293	AAB	65	Fick,1929
S.Africa-blacks	12–14	80	KB	68	Dent, 1937
S.Africa-blacks	10-16	532	Non-verbal	72	Fick, 1939
S.Africa-blacks	8/16	1,008	SPM	75	Notcutt,1950
S.Africa-blacks	Adults	703	SPM	70	Notcutt,1950
S.Africa-blacks	6–10	1,076	DAM	75	Hunkin, 1950
S.Africa-blacks	10–12	278	NVR	74	Lloyd & Pidgeon, 1961
S.Africa-blacks	25	140	WAIS-R	69	Avenant, 1988
S.Africa-blacks	5–13	415	DAM	75	Richter et al., 1989
S.Africa-blacks	9	350	SPM	67	Lynn & Holmshaw, 1990
S.Africa-blacks	16	1,096	SPM	68	Owen, 1992
S.Africa-blacks	15–16	1,093	JAT	68	Lynn & Owen, 1994
S.Africa-blacks	13	49	WISC-R	70	Murdoch, 1994
S.Africa-blacks	17-20	140	SPM	77	Maqsud, 1997
S.Africa-blacks	43	157	WAIS-R/WISC-3	68	Nell, 2000
S.Africa-blacks	16	17	SPM	68	Sonke, 2000
S.Africa-blacks	8	63	WPPSI/ WCST	71	Akande, 2000
S.Africa-blacks	14	152	WCST/WISC-R	65	Skuy et al., 2001
S.Africa-blacks	17	100	WCST/WISC-R/DAM	65	Skuy et al., 2001
S.Africa-blacks	30	196	WAIS-3	82	Claassen et al., 2001
S.Africa-blacks	8–10	806	CPM	68	Jinabhai et al., 2004
S.Africa-blacks	19	711	CPM	71	Vass,1992
S.Africa-blacks	11	379	CPM	71	Knoetze et al., 2005
S.Africa-blacks	6-12	1,333	CPM	71	Linstrom, 2008
S.Africa-blacks	9	340	SPM	69	Malda et al., 2010
S.Africa -colored	10/12	6,196	AAB	83	Fick,1929
S.Africa -colored	13	815	GSAT	86	Claassen,1990
S.Africa -colored	15	778	SPM	80	Owen,1992
S.Africa -Indians	10/12	762	AAB	77	Fick, 1929
S.Africa -Indians	18	284	GFT	88	Taylor & Radford, 1986
S.Africa -Indians	6/8	600	JSAIS	86	Landman, 1988
S.Africa -Indians	15	1,063	SPM	91	Owen, 1992
S.Africa -Indians	15	1,063	JAT	83	Lynn & Owen, 1994
South Africa: median				72

The most recent detailed debate about sub-Saharan intelligence is between Richard Lynn (who gathered the list above) and Jelte Wicherts, who puts the figure closer to 82, but certainly not at European levels. Their argument was basically about the representativeness of samples, particularly to what extent one should include college samples in the national estimates, which will probably be significantly above the local average, and will unfairly boost those estimates.

If you don’t like intelligence scores, then look at PISA 2012 (page 5). http://www.oecd.org/pisa/keyfindings/pisa-2012-results-overview.pdf

In their home countries Asians score in the high 500s and Europeans in the 470 to 520 range. United Arab Emirates 434, Tunisia 388, Qatar 376. No sub-Saharan countries participated. In the past Botswana took part in TIMSS and their score of 364 was very low, but better than that obtained by Kuwait, El Salvador, Saudi Arabia, Ghana and Qatar, so all power to them for having taken part.

As I have observed before, it would appear that as the global economy takes over the world, many nations will be left with both skills shortages and unemployment. Governments are not to blame for everything.

Representativeness beats size

 

Humans are various. If you take a representative sample you find that they differ considerably. If you take a highly selective sample you find that they are more similar. Children admitted to non-selective, fully subsidised schools are a pretty good sample of the population. A very few children may fail to attend because of some infirmity, including severe mental backwardness, and a few may be tutored at home, but otherwise, by and large, the whole range of ability will be fairly represented. Population samples, such as the Scottish surveys give a good picture of ability differences. In crude terms, if you set such an unselected class a task, the fastest child will be about five times faster than the slowest. (We know that to be the case, but not exactly why intellects differ to this significant degree). Any representative sample shows us that intelligence matters.  Unrepresentative samples often suggest that it doesn’t.

What happens when we take a “convenience” sample? Most psychology results are based on college students, because they are convenient to study. That means that they tend to be in in upper half of the ability distribution. The more selective the institution the higher the ability and the narrower the range. In such universities ability differences are diminished. Intelligence becomes less visible because high intelligence was a condition of entry. Other differences in personality and attitudes consequently become more apparent.

The curious impression that intelligence “disappears” after you select for it can lead to many misunderstandings. For example, if you set up a survey on the internet which appeals to clever and diligent people, you tend to get a lot of them replying. “Please take this test of memory and concentration so that we can help people with memory problems”. This will recruit a different sort of person than a request *Rank these 10 celebrities by their sex appeal”. For all I know, the latter task may contribute more to human happiness, but I wager the first group will be brighter, more conscientious, and possibly more likely to be open-minded and liberal in their attitudes. Although most people know about Zimbardo’s “Stanford Prison Experiment” which suggested that ordinary people can become sadistic prison guards, fewer people know that in a replication of the recruiting stages of the experiment the personalities of volunteers for “prison” experiments turned out to be more tough minded than those who applied for the more mundane control experiment. Selection happens even when you don’t know it, and blunt instruments can’t always detect it.

These reflections are engendered by stories that never die. As long ago as 2 January I did a dismissive review of yet another paper trumpeting the finding that IQ does not exist. The popular press love those stories.

http://drjamesthompson.blogspot.co.uk/2013/01/what-makes-good-iq-story.html

The same story is still doing the rounds, like a vampire that does not understand that a stake has been driven through its heart. There are three main problems with the paper. 1) Online intelligence tests attract more intelligent people which reduces common variance and boosts group factors 2) Despite that, the researchers still found a g factor but then obscured that fact by forcing a group factor solution on which they based their conclusions ; and 3) They made very large claims from a selective sample and a selective analysis.

I do not want to give more air time to a questionable interpretation of data, but I do want to draw attention to the fact that representativeness is more important than the total number of persons who take part in a study.  If a million persons take an online intelligence test that is an impressive result but it is very probably far less representative than a properly stratified random sample of 2,400 persons required to re-norm an intelligence test. It might be easy to get a million persons to take part in a quiz about Elvis Presley, including giving an opinion as to whether he was actually dead, but it would be a better assessment of public opinion to select 2000 citizens at random. As to the reality of the mortality of The King, it would probably best to consult one person, the coroner, but that is another matter.

Although I do not value large but unrepresentative samples, is it always pointless to study the effects of intelligence when there is a severe restriction on the range of intelligence? No, not if you know what you are doing as a researcher, and you pay due attention to the effects of restriction of range, and conduct several longitudinal studies so as to check on the stability of your results.

One of the most contested courses in the United Kingdom is medicine. Competition is very tough. It attracts bright people who want to help others. I spent 40 years in their company, hoping that psychology would influence and improve their treatment of patients. Does it matter whether medical students are intelligent or not?

One researcher who took an interest in this matter was Prof Chris McManus, who was in the privileged position of having completed his medical education before turning to psychology. (He also got the 2002 Ig Nobel Prize for Medicine for his 1976 Nature paper on scrotal asymmetry).  He tried to work out what were the best predictors of success in medicine. The last time he told me about his results was in the rarefied company of the few lecturers who taught psychology as applied to medicine at the University of London, about three decades ago.

What are the most recent results derived from long term studies of doctors in the United Kingdom? Do they need to be bright and scholastically able, or are other aspects more important? Should we bother about their exam results?

The Academic Backbone: longitudinal continuities in educational achievement from secondary school and medical school to MRCP(UK) and the specialist register in UK medical students and doctors. IC McManus, Katherine Woolf, Jane Dacre, Elisabeth Paice and Chris Dewberry. BMC Medicine 2013 11:242  http://www.biomedcentral.com/1741-7015/11/242

What McManus has found is that, even in this restricted range of bright persons, there is still an effect of intelligence and educational achievement. Good doctors have to have an academic backbone, and that influences success in their medical careers.

“A-levels correlated somewhat less with undergraduate and post-graduate performance, but there was restriction of range in entrants. General Certificate of Secondary Education (GCSE)/O-level results also predicted undergraduate and post-graduate outcomes, but less so than did A-level results, but there may be incremental
validity for clinical and post-graduate performance. The AH5 (intelligence test) had some significant correlations with outcome, but they were inconsistent. Sex and ethnicity also had predictive effects on measures of educational attainment,
undergraduate, and post-graduate performance. Women performed better in assessments but were less likely to be on the Specialist Register. Non-white participants generally underperformed in undergraduate and post-graduate
assessments, but were equally likely to be on the Specialist Register. There was a suggestion of smaller ethnicity effects in earlier studies.
Conclusions: The existence of the Academic Backbone concept is strongly supported, with attainment at secondary school predicting performance in undergraduate and post-graduate medical assessments, and the effects spanning many years. The Academic Backbone is conceptualized in terms of the development of more
sophisticated underlying structures of knowledge (‘cognitive capital’ and ‘medical capital’). The Academic Backbone provides strong support for using measures of educational attainment, particularly A-levels, in student selection.”

In summary, even when there is a restriction of range in intelligence and scholastic attainment one can show an effect of these variable, and also understand that the restriction of range attenuates that effect. Furthermore, once you have selected for intellect by setting a high bar for entry then the variance accounted for by intelligence in that population will be diminished, but one cannot jump to erroneous conclusions about intelligence no longer being predictive. It is still worth selecting medical student applicants because they are intelligent and have high scholastic achievements.

In summary, it is better to commend a good paper than to lament a weak one.

PISA scores and missing students

 

While I was working very slowly through some of the technical appendices to find how PISA estimates “educational equity” (which turns out to be mean that there is a lack of educational equity anywhere that there is a strong impact of SES on educational performance) Steve Sailer had moved fast into Annex 2a, from which he was able to estimate the representativeness of the samples tested. It looks as if representativeness was often pretty low, which gives some latitude if you decide to get your best students into the exam room, and let the weaker ones be classified as “missing” for the exam day.

http://isteve.blogspot.co.uk/2013/12/pisa-which-countries-not-to-trust.html

 

This looks like being a very important finding. I had spoken vaguely about the possibility of nations “gaming” the test by teaching to the test items, but keeping a good proportion of the weaker students away is a much simpler strategy.

OECD children become OECD adults, eventually

 

The PISA results are out, which give a snapshot of the scholastic attainments of 15 year olds. Only two months ago the findings for adults were published. I cast a baleful eye at that publication in October.

http://drjamesthompson.blogspot.co.uk/2013/10/how-illiterate-is-oecd.html

The OECD found that many adults couldn’t handle intellectual tasks all that well, and as a consequence could not command high salaries.

“The median hourly wage of workers who can make complex inferences and evaluate subtle truth claims or arguments in written texts is more than 60% higher than for workers who can, at best, read relatively short texts to locate a single piece of information. Those with low literacy skills are also more than twice as likely to be unemployed”.

“In all but one participating country, at least one in ten adults is proficient only
at or below Level 1 in literacy or numeracy. In other words, significant numbers of adults do not possess the most basic information-processing skills considered necessary to succeed in today’s world.”

“Across the countries involved in the study, between 4.9% and 27.7% of adults are proficient at the lowest levels in literacy and 8.1% to 31.7% are proficient at the lowest levels in numeracy. At these levels, adults can regularly complete tasks that involve very few steps, limited amounts of information presented in familiar contexts with little distracting information present, and that involve basic cognitive operations, such as locating a single piece of information in a text or performing basic arithmetic operations, but have difficulty with more complex tasks.”

So, we know that the education systems of many countries ten years ago were not turning out uniformly capable citizens, and to the extent that today’s student results are roughly in line with the previous decade’s results, they will not be turning out uniformly capable citizens now. This is because there is a bell curve of ability and because social and because educational systems vary in their effectiveness. Discriminating the relative contributions of these two factors is well nigh impossible unless you take measures of cognitive ability, preferable pre-school, but certainly early in life and including ability at 11 years of age, and then you test their attainments at age 15/16. Basically, if you know what children can do at the end of primary school you are in a good position to see what benefit they get from secondary education. Without those facts, interpretation of educational interventions will be prone to considerable error.

In sum, and as a rule of thumb, not every student who comes out of the education system is able to contribute much to the economy. I have looked at the PISA 2012 results for 15 year olds:

http://www.oecd.org/pisa/keyfindings/pisa-2012-results-overview.pdf

The presentation is in the traditional corporate format: Back to Front. They give you their conclusions, then show you some pictures which buttress their conclusions and then mutter about technical appendices before dissolving into silence. One needs to get into the technical appendixes to get an idea of what is going on. This is somewhat of a mammoth project, and I am almost at the point of helplessly conceding everything, finding myself worn out by the search for the underlying statistical arguments. So, out of self preservation, I will pick out a few problems just lying there, on the surface, and try to come back with more details later. Probably.

The basic requirements of the PISA process: A minimum of 4,500 students from a minimum of 150 schools was required in each country. City states like Singapore were required to sample a minimum of 1,500 students from at least 50 schools. They required the construction of a stratified systematic sample, with sampling probabilities proportional to the estimated number of 15-year-old students in the school based on grade enrolments. Samples were drawn using a two-stage sampling process. The first stage was a sample of schools, and the second stage was a sample of students within schools. Ages were from 15 years 3 months to 16 years 2 months. They were looking for participation rates over 80%. Students with physical or intellectual disabilities were excluded, as were those with insufficient language experience.

If you look at the description of proficiency levels in mathematics and reading, you will find an uncanny similarity with what I described in “The 7 tribes of intellect”. If you look at Exhibit M1 Description of PISA proficiency levels on mathematics literacy scale, at the very simplest level “Students can answer questions involving familiar contexts where all relevant information is present and the questions are clearly defined. They are able to identify information and to carry out routine procedures according to direct instructions in explicit situations. They can perform actions that are almost always obvious and follow immediately from the given stimuli.

Jump to level 3 and Students can execute clearly described procedures, including those that require sequential decisions. [] They typically show some ability to handle percentages, fractions and decimal numbers, and to work with proportional relationships. Their solutions reflect that they have engaged in basic interpretation and reasoning.

There are 6 levels. In fact, if you have a clever silly friend who claims not to understand what intelligence means, get them to read the skill levels, and it will all be spelt out for them. The most able students (level 6) can conceptualize, generalize, and utilize information based on their investigations and modelling of complex problem situations, and can use their knowledge in relatively non-standard contexts. They can link different information sources and representations and flexibly translate among them. Students at this level are capable of advanced mathematical thinking and reasoning.

Equally, from Exhibit R1 Description of PISA proficiency levels on reading literacy scale, at the lowest level “tasks require the reader to locate a single piece of explicitly stated information in a prominent position in a short, syntactically simple text with a familiar context and text type, such as a narrative or a simple list. The text typically provides support to the reader, such as repetition of information, pictures, or familiar symbols. There is minimal competing information.

At level 4 tasks involve retrieving information require the reader to locate and organize several pieces of embedded information. Some tasks at this level require interpreting the meaning of nuances of language in a section of text by taking into account the text as a whole. Other interpretative tasks require understanding and applying categories in an unfamiliar context. Reflective tasks at this level require readers to use formal or public knowledge to hypothesize about or critically evaluate a text. Readers must demonstrate an accurate understanding of long or complex texts whose content or form may be unfamiliar.

And so on. We can describe complexity, but not measure it with theoretically sound techniques. Until we can do that, probably using Kolmagorov complexity (a measure of how difficult it is to fully specify the object in terms of the program size) we are reduced to spelling it all out in ordinary language, which is rather vague.

For example, sending this Mandelbrot sequence as a high resolution picture would take 1.6 megs but it can be generated from a very small computer program a tiny fraction of that size. When we choose a compressed format to mail out a photo we are making use of orderly simplicity. We need to apply that to the classificatiion of cognitive tasks. If you would like to help me do that, please let me know.

Back to PISA 2012. In search of edification, I went to Volume II, which took me a while to find.

“The share of immigrant students in OECD countries increased from 9% in 2003 to 12% in 2012 while the difference in mathematics performance between immigrant and non-immigrant students shrank by 11 score points during the same period.
Immigrant students tend to be socio-economically disadvantaged in comparison to non-immigrant students, yet even when comparing students of similar socio-economic status, immigrant students perform worse in mathematics than
non‑immigrant students. In 2012, they scored an average of 33 points lower in mathematics than non-immigrant students before accounting for socio-economic status, and an average of 21 points lower after accounting for socio‑economic
status. In Canada, Ireland, Israel, New Zealand and the United Kingdom, however, immigrant and non-immigrant students perform equally well.

Comment: Immigrant students are 33 points lower than the locals in mathematics. A drop of this size would reduce the average country result by 33/494 which is 6.7%  PISA have fallen for the sociologist’s fallacy that socio-economic status is entirely imposed externally. That is, that you are poor because the system is stacked against you, rather than that the system responds to how much you work and how much you save.  PISA have “corrected” for this. Some immigrants are poor because they have low skills and low ability. Some immigrants are poor because they have low skills and higher ability but haven’t been allowed to enter an open economy in their home country. Some immigrants have high skills and high ability and are rich. We need better calculations here. Plotting out the immigrant results by years of residence would make the effects easier to understand, as would identifying where these immigrants come from.

“The concentration of immigrant students in a school is not, in itself, associated with poor performance. In general, immigrant students and those who do not speak the language of assessment at home tend to be concentrated in disadvantaged schools. In the United States, for example, 40% of students in disadvantaged schools are immigrants, whereas 13% of students in advantaged schools are. Across OECD countries, students who attend schools where more than one in four students are immigrants tend to perform worse than those in schools with no immigrant students. However, the 19 score-point difference between the two groups is more than halved – to 8 points – after the socio‑economic status of the students and schools is taken into account. ”

Comment: Sociologist’s fallacy again. The give away comes in the initial sentence: “The concentration of immigrant students in a school is not, in itself, associated with poor performance.” This is a disingenuous remark. It is indeed associated with poor performance, but PISA can make that reduce by half by making questionable assumptions.

Across OECD countries, students who reported that they had attended pre-primary school for more than one year score 53 points higher in mathematics – the equivalent of more than one years of schooling – than students who had not attended pre-primary education. In all but two countries with available data, students who had attended pre-primary education for more than one year
outperformed students who had not, after taking socio-economic status into account.

Comment: Sociologist’s fallacy again, plus an unexamined assumption. It might be that pre-school boosts mathematics, but it could just be that brighter parents have brighter children who are ready for school earlier. If we had even a simple IQ measure at rising 4 years of age, like say the Peabody Picture Vocabulary scale, we might be able to understand this better.

In summary, 99% of the press coverage will be based on the summary given by PISA, which contains their own interpretative assumptions. Each country will pore over their own national results. Many will look at the apparent findings and consider implementing them. A very few will get into the detail, and will critique the underlying statistical assumptions.

I expect a high level of critical analysis from the Ho Chi Minh City based publication Vietweek (formerly Thanh Nien Weekly) who sent me detailed questions about the PISA findings for their country, which has done very well, reaching Germany’s level at a fraction of the cost, as Prof Rindermann spotted earlier this year.

I may get into further details about the technical appendices in subsequent posts. Possibly. On the other hand, if you have a couple of research assistants who have spare time, please let me know.

Disclaimer: This post is much, much better when you correct for lack of resources.

Getting value for money in research

 

PISA has been in the news recently. It has a generous budget, a large staff, extensive offices, a skilled press division, and a smooth website, as befits a super-funded OECD organisation. Using all the resources at their disposal, they have come to the following conclusion: Vietnam has a scholastic score (all three scholastic test scores averaged) of SAS 515.67 which is almost identical to Germany whose SAS score is 515.33. So, the scholastic achievements of these two countries are nearly identical, with a very minor advantage for Vietnam, though both are better in scholastic terms than UK at 502.33.
www.oecd.org/pisa/keyfindings/pisa-2012-results-overview.pdf (p. 5)

This result is slightly puzzling, because some previous research by Jim Flynn found a Vietnamese IQ of 94, suggesting scholastic achievements in Vietnam would be lower than in Germany.

Vietnam

12/16

391

SPM

94

Flynn, 1991

In view of this, a researcher who had just published a study in Vietnam a few months ago was quite concerned that the powerhouse of the PISA machine would contradict his own findings. In his paper he had estimated an IQ of 99.43 for Vietnam  compared with an IQ of 99.13 for Germany. According to his calculations both countries were nearly identical, with a very minor advantage for Vietnam, but both were near to the intelligence levels of the UK.

Rindermann, H., Hoang, Q. S. N. & Baumeister, A. E. E. (2013). Cognitive
ability, parenting and instruction in Vietnam and Germany. Intelligence,
41, 366-377.

So, Rindermann got it right. His already published estimates were confirmed by the much larger and very much more expensive PISA study. (Incidentally, all these studies seem to suggest that the UK is punching below its IQ level in scholastic attainments).

How big was his research team? One assistant and a contact in Vietnam, one assistant and him in Germany. If you would like to send him a million dollars his details are below. You might just like to request a copy of his paper, for private study. (Incidentally, it is likely that most research is always done by very small groups of badly funded people, driven by a basic interest in their subject and following research opportunities wherever they arise. Value for money arises from lack of money, and a fundamental interest in facts for their own sake).

In the spirit of full disclosure, I should make it clear that Rindermann and I publish together from time to time, and that he is extremely productive, and very thorough. Teutonic, even. If you want to know what he thinks of American scholastic achievement, try the  chapter on “Flynn Effect in NAEP and narrowing ethnic gaps?” in the December special issue of Intelligence “The Flynn Effect Re-Evaluated”. You might also ask him for a copy of that, or at least a summary.

Prof. Dr. Heiner Rindermann
Professor f. Pädagogische u. Entwicklungspsychologie
TU Chemnitz, Institut f. Psychologie, Wilhelm-Raabe-Str. 43
D-09107 Chemnitz (Deutschland, Germany)
heiner.rindermann@psychologie.tu-chemnitz.de, www.tu-chemnitz.de/~hrin
Professor for Educational and Developmental Psychology

Warning on PISA: Intelligence may be involved

 

Late in the day, I have developed an early warning system to detect when an intelligence related story is entering the public domain: politicians start accusing each other of having fouled up the public education system. The exchange of insults indicates that a set of exam results is coming out, and politicians are playing the blame game. This has now become relatively sophisticated, as these things go, because it includes reference to lagged indicators, and estimates of representativeness and error terms. For example, if current UK 15 year olds are scholastically weak by international standards, is that the fault of the current administration (responsible for the last two years) or the previous administration (responsible for the last 13 years)? Or can we trace the fundamental errors back to the 1870 Education Act? Is our political and journalistic class becoming even more sophisticated? There’s a thesis in that for somebody.

Anyway, last week there was a ritual exchange of insults about the United Kingdom education system which, it was intimated, would shortly be shown by an international study to be “stagnating”. Today the story came out, as trailed, with the PISA results showing that the UK is 26th in the world. League tables seem to have more impact than anything else in British culture, a reference to sport, apparently. Makes it hard to see that a whole bunch of countries are “stagnating” together, within the usual error terms. This is the snapshot most people will see on the news. A more detailed look might follow later.

Here is the BBC graphic:

 

Attention centres on Shanghai, Singapore and Hong Kong who are way ahead, and South Korea, Vietnam and Japan which also do well, but the discussion is all about the methods of education. At no stage in the discussions is it mentioned that, from the point of view of intelligence, these results are entirely predictable.

For example, here are a list of relevant international intelligence test results, based on the comparison group of white Europeans having a “Greenwich Mean” IQ of 100, sd 15. The columns refer to country, age range of people tested, numbers tested, test used, IQ result corrected for Flynn effect as estimated by Richard Lynn (one of the discoverers of the effect), and the name of the author and date of publication. (I am working on getting these into one easily accessible list).

China	6/16	660	WISC-R	107	Dan et al., 1990
China	5/15	5,108	SPM	101	Lynn, 1991
China	14/15	297	Various	103	Li et al., 1996
China	6/12	269	SPM	104	Geary et al., 1997
China	4	60	Arithmetic	109	Ginsberg et al., 1997
China	6/13	463	DAM	103	Cox et al., 1998
China	6/8	160	SPM	107	Goa et al., 1998
China	17	218	SPM	103	Geary et al., 1999
China	19	218	SPM	113	Geary et al., 1999
China	6/8	300	BTBC-R	107	Zhou & Boehm, 2001
China: median				105.5

Hong Kong	9/11	1,007	CCT	105	Godman, 1964
Hong Kong	16	5,209	AH4	106	Vernon, 1982
Hong Kong	10	1,000	SPM	109	Chan & Vernon, 1988
Hong Kong	6/13	13,822	SPM	103	Lynn, Pagliari & Chan, 1988
Hong Kong	6/15	4,500	SPM	110	Lynn, Pagliari & Chan, 1988
Hong Kong	10	197	SPM	108	Lynn, Pagliari & Chan, 1988
Hong Kong	9	376	CCF	104	Lynn, Hampson & Lee, 1988
Hong Kong	9	479	SPM	122	Chan et al., 1991
Hong Kong	15	341	APM	120	Lynn & Chan, 2003
Hong Kong: median				106

Singapore	13	337	SPM	103	Lynn, 1977b
Singapore	15	459	APM	114	Lim, 1994
Singapore: mean				108.5

Korea: South	2/12	440	KABC	113	Moon, 1988
Korea: South	9	107	SPM	109	Lynn & Song, 1994
Korea: South	4	56	Arith	103	Ginsburg et al., 1997
Korea: South	6-16	2,231	WISC-3	100	Georgas et al., 2003
Korea: South: median				106

Japan	5/15	1,070	WISC	102	Lynn, 1977a
Japan	35	316	WAIS	102	Lynn, 1977a
Japan	5/10	760	MFFT	107	Salkind et al., 1978
Japan	10	212	Kyoto	106	Lynn & Dziobon, 1980
Japan	8/11	97	WRAT	108	Tarnopol & Tarnopol, 1980
Japan	9	223	CEFT	112	Bagley et al., 1983
Japan	4/9	347	CMMS	107	Misawa et al., 1984
Japan	6/11	480	Various	105	Stevenson et al., 1985
Japan	6/16	1,100	WISC-R	103	Lynn & Hampson, 1986
Japan	4/6	600	WPPSI	105	Lynn & Hampson, 1987
Japan	14	2,100	Kyoto	104	Lynn et al., 1987a
Japan	13/15	178	DAT	104	Lynn et al., 1987b
Japan	2/8	548	McCarthy	103	Ishikuma et al., 1988
Japan	6/12	142	KABC	101	Kaufman et al., 1989
Japan	16	175	AMM	113	Mann et al., 1990
Japan	9	444	SPM	110	Shigehisa & Lynn, 1991
Japan	5/7	454	CCAT	109	Takeuchi & Scott, 1992
Japan	6/12	451	MAT	105	Tamaoka et al., 1993
Japan	14/15	239	Various	103	Li et al., 1996
Japan	6/17	93	Gen Info	100	Chen et al., 1996
Japan	19	72	GMRT	102	Flaherty, 1997
Japan	7/11	60	DAM	102	Cox et al., 2001
Japan	17	1,119	Gen Info	105	Evans et al. 2002
Japan: median				105

 

So, even if we just consider an IQ test to be a “school far” test (testing things which anyone should know who has been brought up in a particular society), and a scholastic test to be a “school near” test (testing things that are taught at most schools in a particular society) then it is clear that we can test ability early in the life cycle, and use that to predict  scholastic outcomes at 15. If you accept that people vary in intelligence, and that some societies are composed of people who are brighter than people in other societies, then intelligence is part of the mix to be considered when doing international comparisons of scholastic attainments.

And now a few things about PISA: there is no real enforcement of the requirement that the schools tested should be representative of the countries concerned; some countries have strength in depth (appears to be true of China, where even poor provinces do well) while others have much more variability (India), and  it is possible to game the test by teaching to it, rather than to a general curriculum. National exams probably have better coverage of the school population and of the curriculum, but they too can be gamed in various ways, by training children in particular test formats. As a rule of thumb, don’t let the providers of education have the final say about the outcomes of education. Get someone else to judge: universities, employers, patent offices, research labs.

However, whatever the quibbles, the results show that the Far East is about three school years ahead of the West. Sure, the West can ask for a re-count with recent immigrants excluded (PISA does actually provide some of those figures, and even second generation immigrants are usually not at the same standard of the locals), but if you combine a population of IQ 106 with an open economy you get high scholastic attainments, higher than societies with IQs of around 100 and moderately open economies.

Within our own society we probably have 7 tribes of intelligence. The same seems to be true of the world, though we are talking mean differences, not different bands of ability on a single national bell curve. It appears to be the case that the Top Tribe of Global Intelligence reside in China, Japan, Korea and Singapore and thereabouts. (For the moment, we leave to one side small minority groups like European Jews, who are as far ahead of the Chines as the Chinese are ahead of the Europeans). If we take 2011 estimates of a world population of 7.1 billion persons, then with China, Japan, South Korea we have roughly 1.6 billion persons. If they are three school years ahead of the West (three years of full time instruction that will not be easily made up as people take on full time work, so are likely to be permanent) then we (Westerners) have a competitive challenge on our hands.

If we assume that Europe has 740 million inhabitants and the USA 314 then that gives us an estimated Western European population of 1.05 billion. Assuming Easterners have a real IQ of 106 they are 0.65 of a standard deviation above the West. By my crude calculations that implies that only 35% of Westerners are above the Chinese mean, whilst 50% of the Chinese are above their mean, of course.

So, 369 million bright Westerners will be slugging it out against 800 million equally bright Easterners (all of them above GMIQ 106). Western wages are currently about 6 times as high as Chinese wages.

Have I ever explained to you that IQ is only a single number, that there is much more to people than IQ, and that Westerners are very creative, and have multiple intelligences, so many intelligences that, so far, they have never been measured?