bigideas

A recent post at Good Math, Bad Math concerns a proposal for evaluating university courses that tries to measure outcomes. One relatively easy outcome to measure is salary; another is the qualification earned. But, like Mark CC, I started to feel surprisingly skeptical about the whole idea.

Let’s start with money. We’ve said before that we think demand-driven economic considerations — that is, considerations of how economically useful a graduate is to her or his society — give a poor measure of educational utility. I think the supply-side view, which looks at the “utility” of an education to the “customer” who’s “buying” it, is no better.

Using money as a proxy for “utility” doesn’t really work because money itself has utility, and that varies from person to person. Some people like it a lot while other people value other things far more highly (perhaps, even, some of the things they learned about in college). Pointing out that almost nobody would turn down a little extra cash doesn’t go nearly far enough to justify conflating the value of an education with a series of future cashflows.

The problem goes deeper than that, though, and leaves me feeling decidedly Humean about the whole rash of rankings and league tables that erupts whenever a pseudo-market is introduced into a previously non-commercial sector. My worry cuts across a lot more measures than financial ones of the “graduates’ salaries after 5 years” kind.

What you’re looking at when you look at the ranking of an educational institution is a statistical average of a pretty large number of people. This may well be a good predictor of what the outcome will be for a large number of people in the future, if the measure is well-designed and if things don’t change much. The problem is, the number is being used by individuals to make individual decisions. In that context the number gives you almost no useful information.

This doesn’t just affect qualitative outcomes of education like how happy you are, whose measurement is pretty much hopeless. Take an easily-measured outcome like achieving the qualification at the end. Knowing that a shockingly low 21% of the kids at my local comprehensive passed both English and Mathematics GCSEs last year doesn’t tell me whether some particular child will do well or poorly there. You might think that children are interchangeable particles that interact randomly with their education, but if you do you’re mistaken.

My thesis isn’t that league tables tell you nothing, or that quantitative measures of outcomes are worthless. The datum about the GSCE results, for instance, tells me I shouldn’t bet on, say, more than half of the GCSE Maths and English candidates passing next year. It also tells me I should bet on an individual child about whom I have no additional information achieving that result only about one time in five.

The point is, of course, that these are never the circumstances under which ordinary people use this kind of data. They know a great deal of additional information that’s of far greater predictive value than the number in the league table. The number gives us the feeling of making an informed choice, but actually adds virtually no value.

I’m not ideologically opposed to markets, but I am ideologically opposed to people who are ideologically opposed to anything that’s not a market. Giving people the illusion of meaningful choice by giving them the illusion of meaningful information is obviously reassuring. Talking about students as if they were customers and schools as if they were shops is reassuring because that’s a relationship we understand. It’s also a bad analogy that could well lead to harmful policies being pursued by those who find it seductive.