PISA goes to US, finds little bang for buck

 

You may remember that I succumbed to PISA fatigue, and called for dedicated souls to help me churn through the remaining volumes. Andrew Sabisky (@AndrewSabisky) has stepped into the breach, so here is his take on Vol 5 in which PISA visits the US.

PISA 2012 Vol 5

America merits its own volume in the PISA reports, probably because its educational establishment is PISA’s largest customer. Education reformers, backed by the Obama administration, even put together their very own “PISA day” (which fell on December 3rd) to analyze (but not celebrate) America’s PISA results (www.pisaday.org). What message, then, did these ardent educationalists hear from PISA? What messages could they have heard, but, for lack of education, did not receive?

PISA 2012 largely focused on mathematics. America ranks 26th out of 34 participating OECD nations, though it does better in reading (17th) and science (21st). 26% of American 15-year-olds do not reach the PISA “level 2” yardstick of basic mathematical proficiency (OECD average 23%), though this figure falls to 16% after removing those students from an immigrant background. The definition of what counts as “an immigrant background” seems to be given nowhere. Just 2% of American students reach the top band (level 6), compared to 3% across the OECD and “up to 31%” in Shanghai.

These are the basic facts of the American PISA scores. It would seem, therefore, that taken as a group, American students are about average compared to those in other OECD nations, perhaps performing slightly worse in some respects. Of course, treating American students as one unitary group may hide significant and important within-group differences. America is a highly diverse nation, both racially and culturally, though PISA rarely allude to this fact. They do, however, note that Blacks and Hispanics are substantially overrepresented in the American population of low-skilled adults, as measured by PISA’s twin brother, the OECD Survey of Adult Skills (PIAAC). Skills, in the world of PISA, are regarded as all-important, though individual and group differences in the capability of learning certain skills are not considered. Steve Sailer has in fact found data dividing American PISA scores by race with enlightening results (http://isteve.blogspot.co.uk/2013/12/pisa-racial-results-for-americans-on.html).

PISA also attempt to establish some rather less basic and more contentious facts. America apparently has fewer “resilient” students than other countries; just 5% compared to an OECD average of 7%, and between 15-20% in Vietnam, Hong Kong, and Shanghai. “Resilient” students are from the bottom quarter of the socioeconomic status scale (relative to their country of assessment), but who perform in the top quarter of students among all participating countries, after adjusting for socioeconomic status (emphasis added). Therefore, to be classed as resilient, a disadvantaged child does not have to actually score in the top quartile across all nations - he can score considerably less well, but PISA will helpfully bump his scores up. The ghost of the sociologist’s fallacy has returned with a vengeance. The possibility that a low percentage of “resilient” students may in fact function as an indicator of societal meritocracy is also not considered.

Other data, however, perhaps provide more basis for applicable conclusions. The first is that American education is inefficient. Only Norway, Austria, Luxembourg, and Switzerland spend more money per student than America, while nations such as the Slovak Republic spend less than half as much per student as America for the same PISA scores, while Korea also spends considerably less for chart-topping results. Perhaps American educators could safely return some of their budget to the taxpayer and the Gates Foundation, without fear of any loss of performance. There is a statistical relationship between mathematics performance in PISA and national spending on education (r squared = 0.3), though this may be accounted for by the relationship between national IQs and GDP. Cleverer nations, likely to perform better in maths, are also often the wealthier nations on account of their brighter, more productive citizens, and hence have more cash to spend on social goods such as education. PISA, is it worth pointing out, do not follow this line of analysis. America notably spends a higher percentage of its educational budget on “capital outlays” than its competitors (11.4% versus 8.7% OECD average). Perhaps the cash allocated to new school buildings, swanky sports fields, and iPads for all pupils in Los Angeles could be better spent elsewhere.

Does the school climate make a difference? American students are inclined to play truant: 20% have done so within the past two weeks, compared to an OECD average of 15%, and an average of under 5% in many Asian and some European nations. Do American students lack the foresight to make rational decisions in this matter? Perhaps some calculate they can learn more elsewhere? Perhaps some do so accurately. Nevertheless, despite their higher rates of truancy and lower performance, American students do not view their teachers in such a ghastly light; in fact, they view them rather positively, more so than in other OECD nations. 86% of American students think that their teachers are interested in their wellbeing, whereas just 59% of Japanese students do (OECD average 77%). American students also think highly of other aspects of their relationship with their teachers, such as the teachers’ ability to listen and willingness to give their students extra help. The cause of American underperformance cannot seem to be located here. American discipline also seems to be at least on par with the OECD average: 82% of American students report that classroom disruption so severe they cannot work never happens, or almost never happens, compared to an OECD average of 78%. School principals, however, are apparently less sanguine than students about American discipline, though the exact figures are for now lost to PISA’s endless appendices. Overall, the schools with the worst discipline tended to be those with the poorest pupils and the lowest scores, but the nation-level figures perhaps imply here that correlation does not lead us down the path of causation.

School governance is another hot-button topic in nations on both sides of the Atlantic, and here too PISA has some data for policymakers. School autonomy in resource allocation, curricula creation, and assessments apparently correlate positively with higher scores. This relationship persists after controlling for national income, though a correlation coefficient does not appear anywhere. Of course this finding is interpretable in many ways. School autonomy may lead to better results, or more intelligent teachers may demand and receive more freedom from state interference to educate their more intelligent students as they see fit. The usual non-relationship is found (after removing two outliers) between class size and test performance, though Asian nations typically do pay their high school teachers better than America does, whilst also trending towards larger class sizes.

The report also contains a chunky section entitled “Strengths and Weaknesses of American Students in Mathematics”. It runs to 30 pages of A4, but the substance of it is that American students do much worse (relative to Asian and other better-performing nations) on the harder, more cognitively complex PISA items, and do less badly on the easier items. One could interpret this as signifying that some significant amount of the difference between America and the nations that outperform it may be on psychometric g. PISA themselves, however, limit their conclusions to “the relative strengths of the United States lie mostly in the easy items” and “the United States has a particular weakness in the most challenging items”. This is a frustrating failure of analysis just as PISA’s data start to suggest an intriguing pattern worthy of further thought. It is hard to believe that none of the authors of the report had any kind of background in test design.

Lastly, the report attempts to analyze whether or not implementation of the Obama administration’s new “Common Core” universal standards will improve American performance in PISA mathematics. Purely in terms of test readiness, they may indeed do so to a small extent, since the Common Core standards have apparently been heavily drawn from those of PISA. Of course, this assumes the standards are implemented faithfully and without wavering if and when grades begin rapidly to deflate in the face of tougher exams. Experience would perhaps suggest that this is unlikely to occur. We shall see.

Despite its many inadequacies, including a complete failure to include genetic quality (intelligence A) as a variable contributing to both school outcomes and socioeconomic status, the PISA report on American education does retain some value. Most notably it highlights the high rate of financial waste for mediocre results, something the American taxpayer will be displeased to learn. The largely positive atmosphere in American schools does not seem to have much effect on outcomes. For both education reformers and anti-reformers alike, there is much to learn as they fight their never-ending war over the future of schooling in the world’s great power.