OECD children become OECD adults, eventually

 

The PISA results are out, which give a snapshot of the scholastic attainments of 15 year olds. Only two months ago the findings for adults were published. I cast a baleful eye at that publication in October.

http://drjamesthompson.blogspot.co.uk/2013/10/how-illiterate-is-oecd.html

The OECD found that many adults couldn’t handle intellectual tasks all that well, and as a consequence could not command high salaries.

“The median hourly wage of workers who can make complex inferences and evaluate subtle truth claims or arguments in written texts is more than 60% higher than for workers who can, at best, read relatively short texts to locate a single piece of information. Those with low literacy skills are also more than twice as likely to be unemployed”.

“In all but one participating country, at least one in ten adults is proficient only
at or below Level 1 in literacy or numeracy. In other words, significant numbers of adults do not possess the most basic information-processing skills considered necessary to succeed in today’s world.”

“Across the countries involved in the study, between 4.9% and 27.7% of adults are proficient at the lowest levels in literacy and 8.1% to 31.7% are proficient at the lowest levels in numeracy. At these levels, adults can regularly complete tasks that involve very few steps, limited amounts of information presented in familiar contexts with little distracting information present, and that involve basic cognitive operations, such as locating a single piece of information in a text or performing basic arithmetic operations, but have difficulty with more complex tasks.”

So, we know that the education systems of many countries ten years ago were not turning out uniformly capable citizens, and to the extent that today’s student results are roughly in line with the previous decade’s results, they will not be turning out uniformly capable citizens now. This is because there is a bell curve of ability and because social and because educational systems vary in their effectiveness. Discriminating the relative contributions of these two factors is well nigh impossible unless you take measures of cognitive ability, preferable pre-school, but certainly early in life and including ability at 11 years of age, and then you test their attainments at age 15/16. Basically, if you know what children can do at the end of primary school you are in a good position to see what benefit they get from secondary education. Without those facts, interpretation of educational interventions will be prone to considerable error.

In sum, and as a rule of thumb, not every student who comes out of the education system is able to contribute much to the economy. I have looked at the PISA 2012 results for 15 year olds:

http://www.oecd.org/pisa/keyfindings/pisa-2012-results-overview.pdf

The presentation is in the traditional corporate format: Back to Front. They give you their conclusions, then show you some pictures which buttress their conclusions and then mutter about technical appendices before dissolving into silence. One needs to get into the technical appendixes to get an idea of what is going on. This is somewhat of a mammoth project, and I am almost at the point of helplessly conceding everything, finding myself worn out by the search for the underlying statistical arguments. So, out of self preservation, I will pick out a few problems just lying there, on the surface, and try to come back with more details later. Probably.

The basic requirements of the PISA process: A minimum of 4,500 students from a minimum of 150 schools was required in each country. City states like Singapore were required to sample a minimum of 1,500 students from at least 50 schools. They required the construction of a stratified systematic sample, with sampling probabilities proportional to the estimated number of 15-year-old students in the school based on grade enrolments. Samples were drawn using a two-stage sampling process. The first stage was a sample of schools, and the second stage was a sample of students within schools. Ages were from 15 years 3 months to 16 years 2 months. They were looking for participation rates over 80%. Students with physical or intellectual disabilities were excluded, as were those with insufficient language experience.

If you look at the description of proficiency levels in mathematics and reading, you will find an uncanny similarity with what I described in “The 7 tribes of intellect”. If you look at Exhibit M1 Description of PISA proficiency levels on mathematics literacy scale, at the very simplest level “Students can answer questions involving familiar contexts where all relevant information is present and the questions are clearly defined. They are able to identify information and to carry out routine procedures according to direct instructions in explicit situations. They can perform actions that are almost always obvious and follow immediately from the given stimuli.

Jump to level 3 and Students can execute clearly described procedures, including those that require sequential decisions. [] They typically show some ability to handle percentages, fractions and decimal numbers, and to work with proportional relationships. Their solutions reflect that they have engaged in basic interpretation and reasoning.

There are 6 levels. In fact, if you have a clever silly friend who claims not to understand what intelligence means, get them to read the skill levels, and it will all be spelt out for them. The most able students (level 6) can conceptualize, generalize, and utilize information based on their investigations and modelling of complex problem situations, and can use their knowledge in relatively non-standard contexts. They can link different information sources and representations and flexibly translate among them. Students at this level are capable of advanced mathematical thinking and reasoning.

Equally, from Exhibit R1 Description of PISA proficiency levels on reading literacy scale, at the lowest level “tasks require the reader to locate a single piece of explicitly stated information in a prominent position in a short, syntactically simple text with a familiar context and text type, such as a narrative or a simple list. The text typically provides support to the reader, such as repetition of information, pictures, or familiar symbols. There is minimal competing information.

At level 4 tasks involve retrieving information require the reader to locate and organize several pieces of embedded information. Some tasks at this level require interpreting the meaning of nuances of language in a section of text by taking into account the text as a whole. Other interpretative tasks require understanding and applying categories in an unfamiliar context. Reflective tasks at this level require readers to use formal or public knowledge to hypothesize about or critically evaluate a text. Readers must demonstrate an accurate understanding of long or complex texts whose content or form may be unfamiliar.

And so on. We can describe complexity, but not measure it with theoretically sound techniques. Until we can do that, probably using Kolmagorov complexity (a measure of how difficult it is to fully specify the object in terms of the program size) we are reduced to spelling it all out in ordinary language, which is rather vague.

For example, sending this Mandelbrot sequence as a high resolution picture would take 1.6 megs but it can be generated from a very small computer program a tiny fraction of that size. When we choose a compressed format to mail out a photo we are making use of orderly simplicity. We need to apply that to the classificatiion of cognitive tasks. If you would like to help me do that, please let me know.

Back to PISA 2012. In search of edification, I went to Volume II, which took me a while to find.

“The share of immigrant students in OECD countries increased from 9% in 2003 to 12% in 2012 while the difference in mathematics performance between immigrant and non-immigrant students shrank by 11 score points during the same period.
Immigrant students tend to be socio-economically disadvantaged in comparison to non-immigrant students, yet even when comparing students of similar socio-economic status, immigrant students perform worse in mathematics than
non‑immigrant students. In 2012, they scored an average of 33 points lower in mathematics than non-immigrant students before accounting for socio-economic status, and an average of 21 points lower after accounting for socio‑economic
status. In Canada, Ireland, Israel, New Zealand and the United Kingdom, however, immigrant and non-immigrant students perform equally well.

Comment: Immigrant students are 33 points lower than the locals in mathematics. A drop of this size would reduce the average country result by 33/494 which is 6.7%  PISA have fallen for the sociologist’s fallacy that socio-economic status is entirely imposed externally. That is, that you are poor because the system is stacked against you, rather than that the system responds to how much you work and how much you save.  PISA have “corrected” for this. Some immigrants are poor because they have low skills and low ability. Some immigrants are poor because they have low skills and higher ability but haven’t been allowed to enter an open economy in their home country. Some immigrants have high skills and high ability and are rich. We need better calculations here. Plotting out the immigrant results by years of residence would make the effects easier to understand, as would identifying where these immigrants come from.

“The concentration of immigrant students in a school is not, in itself, associated with poor performance. In general, immigrant students and those who do not speak the language of assessment at home tend to be concentrated in disadvantaged schools. In the United States, for example, 40% of students in disadvantaged schools are immigrants, whereas 13% of students in advantaged schools are. Across OECD countries, students who attend schools where more than one in four students are immigrants tend to perform worse than those in schools with no immigrant students. However, the 19 score-point difference between the two groups is more than halved – to 8 points – after the socio‑economic status of the students and schools is taken into account. ”

Comment: Sociologist’s fallacy again. The give away comes in the initial sentence: “The concentration of immigrant students in a school is not, in itself, associated with poor performance.” This is a disingenuous remark. It is indeed associated with poor performance, but PISA can make that reduce by half by making questionable assumptions.

Across OECD countries, students who reported that they had attended pre-primary school for more than one year score 53 points higher in mathematics – the equivalent of more than one years of schooling – than students who had not attended pre-primary education. In all but two countries with available data, students who had attended pre-primary education for more than one year
outperformed students who had not, after taking socio-economic status into account.

Comment: Sociologist’s fallacy again, plus an unexamined assumption. It might be that pre-school boosts mathematics, but it could just be that brighter parents have brighter children who are ready for school earlier. If we had even a simple IQ measure at rising 4 years of age, like say the Peabody Picture Vocabulary scale, we might be able to understand this better.

In summary, 99% of the press coverage will be based on the summary given by PISA, which contains their own interpretative assumptions. Each country will pore over their own national results. Many will look at the apparent findings and consider implementing them. A very few will get into the detail, and will critique the underlying statistical assumptions.

I expect a high level of critical analysis from the Ho Chi Minh City based publication Vietweek (formerly Thanh Nien Weekly) who sent me detailed questions about the PISA findings for their country, which has done very well, reaching Germany’s level at a fraction of the cost, as Prof Rindermann spotted earlier this year.

I may get into further details about the technical appendices in subsequent posts. Possibly. On the other hand, if you have a couple of research assistants who have spare time, please let me know.

Disclaimer: This post is much, much better when you correct for lack of resources.