Is PISA arguing equitably?

 

I have been working through Volume II of PISA 2012, and I now have less idea of what is going on than when I started. The OECD technique is to use a version of visual Chi-square, and plod through every country’s results one variable at a time, depicted on a four by four table. The underlying statistics are not given. The correlations between measures are not given. Even when they plot out the data in a traditional scatterplot and draw a line of best fit (Fig 11.2.1) no correlation is given. Figure 11.2.11 is a good example. There are 11 categories plotted out for each country. It appears that Luxembourg has few average schools, Finland has many. I cannot work out what point is being made, if any. These data call out for a simple correlation matrix, and some simple regression equations, with attainment being predicted by all the main variables. Researchers seem to be kept at arm’s length while the drum roll of national comparisons surge across page after page of the report. “Concentrate on our conclusions” seems to be the message, “and leave the details till later, far later.”

Seeking clarity wherever I could find it, I read one of the key conclusions in Vol II, which is that learning opportunities must be distributed equitably in society. Here is their argument:

“In OECD countries, parents of socio-economically advantaged students (those in the top quarter of the socio-economic distribution in their country, or one standard deviation above the average on the PISA index of economic, social and cultural status) are highly educated (95% have attained a tertiary education) and work in skilled occupations (97%). In contrast, the parents of socio-economically disadvantaged students (those in the bottom quarter of the socio-economic distribution in their country, or one standard deviation below the average on the PISA index of economic, social and cultural status), have much lower educational attainment, and very few (6%) work in skilled occupations. Advantaged students also report having many more books at home than their disadvantaged peers (282 compared with 69 on average), as well as works of art, classical literature and books of poetry (Table II.2.2). While disadvantaged students have fewer books, cultural possessions and some educational resources at home, a large majority has access to a desk, a quiet place to study, a dictionary, a computer and an Internet connection at home (Table II.2.2). For a more detailed definition of socio-economic advantage and disadvantage, as measured by PISA, see Box II.2.1 below.

Large differences in performance associated with the background of students and schools – whether socio-economic status, immigrant or language background – signal that learning opportunities are not equitably distributed throughout a school system or that not all students have access to the high-quality instruction and material, financial and human resources that could help them succeed in school and beyond.”

Of course, there is another interpretation of these findings, which is that bright parents have better jobs, save more money, and give birth to brighter children. A good proportion of the variance could be genetic. Indeed, Robert Plomin and his team have shown this to be the case. Books do not magically boost intelligence. They have to be read. That means that the parents have got to want to read to their children, to encourage them to read, to talk about what they have read, and to suggest further books to read. A proper study would consider that possibility, and attempt to test the power of the conflicting interpretations. There is also an assumption that home environments are de facto part of the school system, and that if a family supports their children they are also contributing to inequity.

One way to look at this claim would be to show the correlations between attainment and social variables, but I cannot find these as I work through Volume II.

PISA continues:

What the data tell us

• Some 6% of students across OECD countries – nearly one million students – are “resilient”, meaning that they beat the socio-economic odds against them and exceed expectations, when compared with students in other countries. In Korea, Hong Kong-China, Macao-China, Shanghai-China, Singapore and Viet Nam, 13% of students or more are resilient and perform among the top 25% of students across all participating countries and economies.

• Across OECD countries, a more socio-economically advantaged student scores 39 points higher in mathematics – the equivalent of nearly one year of schooling – than a less-advantaged student.

For some reason, “resilience” is very similar to “intelligent even though relatively poor” and is most frequent among people of Far Eastern descent. The OECD has assumed that wealth and social status leads to scholastic success. Then they find that their theory has a significant hole in it. Rather than re-thinking their theory, they keep the theory but argue that some people are exceptions because of some mystical “resilience”. It would be better to ask more generally why some people are studious and intelligent, and then see how that relates to other variables.

Also, consider for a moment why SES should be related to scholastic outcome. One possibility, the one favoured by PISA, is that differences in SES are due to the unfair allocation of resources in a society. The rich get all the pleasure, the poor get all the blame, and at the very least there should be compensatory spending within the education system to make up for this gross unfairness. The other explanation is that a society has achieved social mobility. The brightest have risen to the top, though they may not remain there if they lack the ability to do so. At any one time there will be some people who are well educated and rich, though they may not be exactly the same people 15 years later.

For example, one way to understand what is happening is to try to distinguish between the wealth of the well-educated, and the education/intelligence of the well-educated. Is it better to be well-educated and poor than less well educated and rich?

Heiner Rindermann looked at this in 16 cross-sectional and 3 longitudinal samples in six countries (USA, Austria, Germany, Costa Rica, Ecuador, Brazil) and analyzed the relative impact of parental education compared to wealth on the cognitive ability of children (aged 2 to 23, total N=15,325). The social background ranged from welfare recipients and poor indigenous people in remote villages to professional (academic) families in developing and developed countries. Children’s cognitive ability was measured with different tests (mental speed tests, CFT, the Ravens, Stanford-Binet, PPVT-R, WET, CogAT, Piagetian tasks, ASVAB, PIRLS, TIMSS, PISA). Parental wealth was estimated through questionnaires by directly asking for income, indirectly by self-assessment of wealth compared to others, and by evaluating assets. Parental education comprised school and professional education. The mean direct effect of parental education was b_Ed=.40, of income/wealth b_In=.16 (N=15019, k=16 samples). In all path analyses parental education showed a stronger impact on intelligence than economic status (total effects: b_Ed=.45 for education, b_In=.12 for wealth, N=15019, k=16). Further important factors for children’s cognitive ability depending on parental education are number of books (b_Bo=.18), marital status (b_CF=.17), educational behavior of parents (b_EB=.12) and behavior of children themselves (b_BC=.19). So, books do have an effect, and so does marital status and parent’s helping their children, but all those are not as powerful as parent’s education. Of course, parent’s education is related to parent’s intelligence, which could not be measured directly in these samples.

Possibly, somewhere in these many volumes of PISA results this very calculation has been done. Please help me find it. If it is not there, we should ask them to work on it, because it would be very interesting to see if their very large sample replicates what Rindermann found.

I continue to plod through the Volumes, and have now got a helper in the OECD library to guide me to the bits I want to see. Will try to keep you posted.