Mr Popper's Predictions

In My Experience

In my experience whenever you hear someone saying "in my experience" you're about to get an earful of incoherent nonsense justified by the observer's single perspective. It's nearly always dangerous nonsense, justified by specific examples taken from a single snapshot in time.

Well, in my experience, personal observations are typified by overconfidence, colored by hindsight bias and impervious to evidence suggesting that they're wrong. They're flung about with gay abandon, but have as much in common with objective truth as a report from an analyst.  The future is unknowable, anyone who claims special knowledge is either lying or mad. And possibly both.

Twaddle

A lot of people, including investors and advisors, spend a lot of time talking unmitigated twaddle, repeating popular memes without any real evidence based on either some level of claimed expertise or, more often I suspect, simply regurgitating anecdotes as though they're objective reality. Of course some people claim they don't believe in objective reality either, but I bet they don't regularly test their beliefs by leaving their homes through the second floor window.

Most of the useful knowledge in the world has arisen out of the use of the scientific method. This is even true in social sciences like economics, where we can learn more by careful studies than we can by any number of anecdotes.  For example, see the work of John List and Uri Gneezy that we discussed in Beyond the Dismal Science, who use naturally occurring situations to run field experiments, which they can use to test various economic hypotheses.

Disconfirmation

The scientific method enshrines a particular and peculiar way of working because at its heart it relies on refutation, not proof. Scientists are usually to be found trying to disprove something some other scientist thinks they've proven. As the philosopher of science Karl Popper observed, it's a gigantic exercise in disconfirmation.

It works like this: (1) come up with an idea (the hypothesis); (2) come up with an experiment to test the hypothesis (if you can't come up with an experiment the hypothesis is wrong, or at least not worth testing), (3) see if the results of the experiment prove the hypothesis, (4) publish your findings and watch as everyone else tries to find the flaws in what you've done and (5) if they can't and you live long enough and don't die you pick up a Nobel Prize.

Although that last bit is discretionary.

Popper

In Popper's philosophy this is a method of reasoning known as modus tollens:

If the theory is true then the prediction is true

The prediction is not true

Therefore, the theory is not true

The point about this is that the scientific method enshrines a mechanism to invalidate its own findings, if evidence arises to discredit them. Research findings are published, reviewed, replicated, tested, probed - and sometimes revealed to be wrong or, more often, only a partial truth.

Sadly, despite this, anecdotes remain a popular way of disseminating spurious bits of knowledge even though ten stories constitute no more evidence than one and a hundred no more than ten. But people don't think in terms of the scientific method most of the time, even those that understand it. Mostly people don't think about science at all, other than to think it's all a bit complicated and to suspect that scientists often make things up to sound important.

Foundational

But of course the proof of science is all around us. Scientific achievements build on each other, until we have entire citadels of ideas, and there comes a point where even the most cynical scientists will accept that some foundational hypotheses have to be true: gene therapy is based on evolution, computers on quantum theory, GPS on relativity; and so on. People believed in electricity when Edison popularised the light bulb, not when Faraday figured out how to generate a current.

The problem is that although personal experience can incline individuals to strong beliefs about things they don't necessarily establish any eternal - or even transitory - truths. Even if thousands of people believe in something like homeopathy or intelligent design or technical analysis it doesn't prove that it works. After all historically astrology was widely believed - it still is if you believe the evidence of Google - but it doesn't make it true.

Kudos

All too often people confuse the scientific process with scientists, but scientists are as flawed and human as the rest of us, as subject to confirmation bias and hindsight bias as anyone else. But scientific ideas are part of a market in which the currency is knowledge - and overturning the established verities with new knowledge is the incentive driving researchers: there's no fame or fortune in proving that which is already known, but there's lots of kudos to be acquired by making someone famous look stupid.

And thus, out of the very fallibilities that make us human and untrustworthy, we've developed a method which continually evaluates and reinvents itself, an engine that's driven us from subsistence farms to server farms in three centuries. Perhaps the most visible sign of the success of the scientific method is in life expectancy which has soared in the developed world. In South Korea, perhaps the most recently fully industrialised country, the average lifespan has increased by 27 years in the last 50.

Evidence

Now, frankly, if science lies behind all of these wonders you'd think that we, as reasonably intelligent apes, would think through the implications of this and start to dimly recognise that maybe our intuitions and personal experiences might not be the best guides to our future success. But generally we act as though the miracles and wonders that surround us are acts of some benevolent deity rather than the crowning achievements of the human race, and continue to behave as though our own, peculiarly subjective, views are worth more than any amount of proper evidence based analysis.

Which is why those three little words "In My Experience" are the harbinger of great danger: because the person uttering them, or writing them, is convinced that their personal experience offers some useful knowledge for other people.  And almost always it doesn't. In fact it's often not clear that the speaker has learned anything very useful from their own subjective perspective, let alone derived any useful objective knowledge that the rest of us can adapt for our own uses.

Credulous

Investors, in general, would do well to act more like scientists and less like credulous apes. The field experiments conducted by List and Gneezy offer one way of doing this, developing hypotheses and then testing them - all too often investors ignore the results of their decisions and fail to measure their results. The other way is to take advantage of the vast amounts of research done for them - often based on statistical analysis of the behavior of lots of similarly motivated investors.

The thing is, you see, that in my experience we are all much more similar than we'd like to believe. Fortunately you don't have to take my word for that, the research is out there, if you want to go and find it.

  

Related articles:

• Beyond the Dismal Science
• Newton's Financial Crisis: the Limits of Quantification
• Investing By Jerks
• Science, Stocks and Superstition
• Whither Forecasting? The Butterfly Stirs ...
• Perpetual Novelty, Santa Fe Style
• Money Matters, Your Opinion Doesn't