Debating Statistics

Russell Roberts over at Cafe Hayek and Robin Hanson over at Overcoming Bias argue about the value of statistical techniques. Roberts:

The nature of the analysis is such that neither side can convince the other that "their" analysis is reliable. That's not always true. As I suggest in the podcast, Milton Friedman was able to convince the skeptics that inflation is everywhere and always a monetary phenomenon. Friedman won the debate. But how many other studies can you think of where someone staked out a controversial position and convinced the skeptics based on empirical analysis? I think it can be done, but it's rare. And in today's world, most of the interesting empirical claims are being made in cases where the data are too incomplete and the issue is so complex that we can't move to a consensus. The empirical work doesn't improve our understanding of what's going on. It masks what's going on. It gives a patina of science when in effect the numbers aren't really informing the debate.

Hanson:

If Russ relies little on data to draw his conclusions, then on what does he rely? Perhaps he relies on theoretical arguments. But can't we say the same thing about theory, that we mainly just search for theory arguments to support preconceived conclusions? If so, what is left, if we rely on neither data nor theory?

Try saying this out loud: "Neither the data nor theory I've come across much explain why I believe this conclusion, relative to my random whim, inherited personality, and early culture and indoctrination, and I have no good reasons to think these are much correlated with truth." That does not seem a conclusion worth retaining.

Roberts:

My basic point was that when it comes to high-powered sophisticated statistical techniques, our biases as researchers and as consumers of that research often triumph over truth. The truth is elusive in complex systems with many things changing at once. It’s hard to isolate the independent effect of one particular variable. When scholars can run hundreds of multivariate regressions at very low cost, it easy to convince yourself that the results that confirm your prior beliefs are the “right “ results. The ones that failed must be the “bad ones.”

[H/t: Saifedean Ammous]