Faking good on PISA

 

 

One of the delights of being a member of a community of researchers in the modern age is the speed with which colleagues can come together to answer a question and scope out a solution to a problem.

Steve Sailer has looked at the most recent PISA results, which he has been discussing generically for many years. 

http://www.unz.com/isteve/new-pisa-test-results/

He pointed out that in some countries a large proportion of eligible children don’t show up in the statistics. Could it possibly be the case that they are discretely told to stay at home, because national pride is at stake? Perish the thought! He pointed out that Argentina had apparently made stellar gains, but a commentator on his blog pointed out later that there was so much cheating in the Argentine provinces that the results had to be discarded, and the declared results are for Buenos Aires only, so probably higher than the national figures, or so the porteños would have you believe. Incidentally, it is only recently that Argentina has had economic data, such as for inflation, that could be vaguely trusted, so they are only just in the Truth Recovery phase.

Cheating is the easiest way to boost results. Teachers can look at the questions some days before the test, and do a crash course in “revision” for the class. This makes teachers, children, parents and governments happy. PISA says it has methods to ensure security and detect cheating, but Heiner Rindermann also has his own ability to look carefully at PISA’s published results, and rejects some of them on the grounds of improbability.

Anatoly Karlin also had a look at the dataset and discussed the disappointing performance of China and other eastern countries, with Russia doing better. Get his full account here:

http://www.unz.com/akarlin/pisa-2015/

I wondered how big the effect of such selective non-attendance on the examinations might be. There is also the confounder that age at ending secondary education varies between one country and another, so that must be factored into the equation.

Emil Kirkegaard suggested an approach, and after discussions with me and Gerhard Meisenberg, sorted it out quickly. Have a look at the full process here:

http://rpubs.com/EmilOWK/233323

Emil had also asked Heiner Rindermann to comment, and he came in a few minutes later, with a detailed publication (not yet published, so I cannot show it to you) and a rule of thumb adjustment you can apply to all the countries.

Heiner says:

School attendance rate of 15 year old youth (usually, but not always,  given in PISA reports, usually somewhere at the end).
Do not confuse with participation rate in PISA study.

Per each percent point not attending school subtract 1.5 SASQ points (equivalent 0.225 IQ points). That is a rule of thumb.

I have made a smaller correction for countries at low ability levels - in such countries pupils in school do not learn much.

Not bad for a few hours of internet time.

A few hours later, Steve Sailer had further and better particulars on the results:

http://www.unz.com/isteve/2015-pisa-mean-scores-in-perspective/

So, where does this leave us with the PISA results? First, it gives me a chance to quote myself, one of the consolations of a lonely blogger: “Nobody gets round sampling theory, not even the Spanish Inquisition.” 

Second, and arising from the quote, the consequence is that the PISA results are only generalizable if the sample is a fair selection of the relevant group. In my view, to understand the abilities of a nation, the relevant group should be the entire age cohort. If many 15 year olds have already left school then a school sample will always be a partial indicator of a nation, and will very probably flatter it. This is because weaker students find school frustrating and leave, whereas the brighter ones enjoy studying, understand its long term benefits, and stay in education as long as they can. Further, if teachers ensure that even among those still staying at school the weaker students fall discretely ill on the day of testing, then the results can be massaged upwards. Spotting weaker students is easy for teachers: they can quickly determine it from student questions, and more accurately determine it by marking their class test papers.

Third, I do not want to reject PISA results, because local examination results share many of the same problems. In any nation where some teenagers leave school early the local examination results will be better than the actual national average. Equally, if within a school cohort not everyone takes the same national examination, the same flattering distortion takes place.

Fourth and finally, I think it best to study PISA results once they have been corrected to account for incomplete age cohorts in the Rindermann fashion, or in some elaboration and refinement of that technique.  Absent that, they have a large error term and present too rosy a picture of national scholastic attainments.