Eyes Above The Waves

Robert O'Callahan. Christian. Repatriate Kiwi. Hacker.

Tuesday 3 December 2013

Another Knee-Jerk Reaction To International Rankings

Predictably the OECD "Pisa" report ranking countries' education results has caused a stir in New Zealand. New Zealanders, or at least their news media, love international rankings of all kinds --- especially if they can be portrayed negatively for New Zealand. As often, the latest report, and the discussion around it, has some major problems.

For starters it's interesting to compare the initial NZ Herald story with the more nuanced reporting from the AP wire story. The Herald chose the completely fallacious headline Significant drops in NZ educational achievement --- fallacious because a drop in ranking does not necessarily mean a drop in actual achievement (and in this case, there is no evidence of a drop in achievement). (PPTA president Angela Roberts gets this right here.) The Herald story (and its followups) simply ignores the issues with the Pisa report which are touched on by the AP story, and better explained in Slate (mainly, it's invalid to compare city-states with entire countries, especially including Shanghai but no other part of China!).

Apart from that, some of the reactions to the report are ridiculous. There's this:

But Labour says any drop in the rankings should be sheeted home to an excessive focus by National on "testing" over the past five years. Although National Standards does not actually involve national testing, Labour's education spokesman Chris Hipkins said, "It shows that the last five years' focus on test-taking has been a disaster and it has actually narrowed the focus of our system and it has actually decreased the level of achievement within the education system".
NZ's ranking drop is mainly due to Asian countries increasing theirs. Those countries' education systems are far more focused on testing than NZ's has ever been, which is probably why their ranking is increasing: if you focus on testing (and teach to the test), you do really well on tests, which is of course what studies like Pisa measure, since tests produce data and other educational activities don't. Chris Hipkins, if you really think Pisa is important you should advocate a big increase in national testing.

I actually think national standardized testing is important, but it's not the only important thing and the Asian education systems ranked highly in the Pisa report have massive problems despite producing good test results. My wife and many other people I know went through those systems and describe how they're focused on rote memorization and discourage any kind of learning other than school and after-school coaching on their core subjects. For example, I taught myself computer programming in my copious spare time after school, but for most children in Hong Kong that simply wouldn't have been possible. People talk about how in exams you "give it back to the teacher" --- cram for exams, do well, and then forget it as you prepare for the next one. I think it would be very interesting to re-test children three years after they left school to see what they've retained.

It's really important to avoid over-optimizing for the things we can measure at the cost of the things we can't as easily measure. It's also really important to not overreact to every international ranking report. We have to think critically about these things. I wish the media would.