Regression to the Mean: Of Nazis and Investment Analysis

Francis Galton, The Measuring Man

Francis Galton, cousin of Charles Darwin, was utterly uninterested in the both the workings of investment analysts and Hitler’s future plans for world domination. To be fair he died before both really got going. However, he has the dubious distinction of being at least partially to blame for some of the madder excesses of both.

What Galton was interested in was measuring things: mainly people, although he’d turn his hand to anything in the absence of a likely suspect or two. In so doing he uncovered a statistical phenomenon known as Regression to the Mean which lies behind many of the theories which still dominate stockmarket analysis and valuation techniques today and which many people misunderstand totally.

Galton’s Eugenics

To deal with Galton and Hitler first, however. Galton was the inventor of eugenics, a branch of science dedicated to analysing the fitness of the human species with, as its goal, the aim of improving the race. Like many a Victorian gentleman Galton was convinced that humanity was regressing into a less intelligent and less developed state. In fact what he was actually observing was the impact on people of moving them into unhygienic cities, working them in appalling conditions and giving them a lousy diet. Tricky stuff this nurture, nature thing.

Galton’s measurements were essentially designed to show that the rich and powerful were clever and the poor and weak were not, thus squaring the circle: the rich and powerful were naturally bound to ascend to the top of the tree and therefore had every right to be there. Nothing at all, then, to do with the fact that your great-great-great-grandmother had an illegitimate child with the King, was set up with an estate comprising half of Scotland in order to go away and stop bothering him and you inherited it by accident of birth. Clearly not.

Galton’s Statistics

Although Galton’s logic may have been a little faulty when it came to measuring such subjective things as intelligence, when he turned his attention to more objective stuff like height there wasn’t much wrong with his statistics. In a set of experiments on the size of seeds of successive generations of sweet peas he noted the odd fact that the larger seeds of one generation tended to result in smaller seeds in the next generation and vice versa. Applying a similar analysis to people he noted that although short people tended to have short children and tall people to have tall children the short children tended to be slightly taller than their parents and the tall children slightly shorter than theirs.

This led Galton to postulate the principle known as Regression to the Mean. What this says is that is an abnormally large value of a variable – let’s say it’s the return on the stockmarket in any given year – is likely to be succeeded by a lower value. You could say that it’s the statistical proof of the old saving “what goes up must come down”. However, what Galton also showed was that the results of such analyses tended to cluster around the mid-point – the so called mean. The average value if you like. Over time it’s this average value that all returns will converge on. Technically this convergence is called “regression”. So values converge on the average or “regress to the mean”.

Think about tossing a coin ten times. On average you’ll get five heads and five tails. If last time you got ten heads next time you’re more likely to approach the average value or regress to the mean. It’s just probability.

Stockmarket Returns and Sheep

Over the twentieth century the stockmarket returned, roughly, 12% per year. Most years saw a return a bit more or a bit less than this value. A few years saw a return much, much, much higher than this. A few saw returns much, much, much lower than this. What Galton’s findings suggest for stockmarket investors is that worrying over (or glorying in) exceptionally bad (or exceptionally good) years is pointless. Over time things will drift back to the mean. Over time, if you don’t let better judgement get in the way, you would have made about 12% a year.

Often, though, judgement does get in the way, not always for the better. There are certainly a few people who have done better than 12% over very long periods of time – too long to be explained by chance. There are lots and lots of people who’ve done worse. In both cases the explanation for the exceptional performance probably lies in psychology: the jitterings of markets mean that at both the high and low points there are opportunities for discerning investors to make cool judgements - sell at the highs and buy at the lows. Most investors, though, aren’t discerning. They’re just humans and they do what humans do. They flock like sheep. Baa.

So they buy when everyone else is buying and sell when everyone else is selling. Buy high, sell low is not textbook mantra but it’s done by far more people than do the opposite. In the sixties it was the Nifty Fifty, in the nineties it was dotcom – sooner or later it’ll be something else. The herd effect is well ingrained in our behaviours and if everyone else is doing something it stands to reason that we should be doing it to. Obviously. Baa.

Neither Students nor Stockmarkets are Normal

There are two things about mean regression that investors need to be careful of. Firstly the stockmarket doesn’t follow exactly the same distribution as naturally occurring phenomena (which technically is called a Normal or Gaussian distribution). So we get very bad years and very good years far more than the theory would predict. Over time returns will still tend to the mean, but we need to be aware that the markets may trend away from this for considerable periods of time. Simple psychology makes many investors extrapolate current conditions forever which is not sensible and is likely to lead to exactly the wrong sorts of behaviour. Clever academics can make the same mistake (see Alpha and Beta - Beware Gift Bearing Greeks).

Secondly, there’s a problem with mean regression which is that it’s a statistical effect. If you’re a below average student then you can assume you’ll get a below average mark in a test. You won’t get any closer to the mean on a second test just because of a statistical effect: it may be that you were unlucky to get an exceptionally low mark first time out and therefore you’ll do better next time but that’s not statistics, that’s you.

Stats Doesn’t Mean You Can Buck the Trend

The same applies to companies in the absence of any changes. Just as a low grade student will predictably score low grade marks so a low grade company will predictably get low grade results. However, very often underperforming companies will get shaken up by new management or shareholder discontent in much the same way as a student’s father may discipline them to get them to work harder. But this doesn’t happen by chance, which means the effort of stock analysis is not irrelevant and that simply picking a sample of underperforming companies won’t guarantee excess returns.

Although Galton observed mean regression in people he fundamentally misinterpreted it in the light of his own prejudices. Whereas we now know that regression is a statistical phenomenon where any lack of full correlation between parents and children will result in such an effect (i.e. children are not copies of their parents due to gene mixing and mutation) Galton attributed it to a combination of inheritance from parents and from prior ancestors. Hence he was able, or so he thought, to demonstrate that the inherited characteristics of the privileged flowed down from their distinguished lineages. This fundamental misunderstanding of the phenomena was the mainspring of the eugenics movement which ultimately led to Hitler and his death camps.

Statistics in the wrong hands is very, very dangerous. Losing money on the stockmarket because of it is possibly the least of our worries.



Related Posts: Risky Bankers Need Swiss Cheese Not VaR, Alpha and Beta: Beware Gift Bearing Greeks