The Trouble with Models

The economy, political events and even the sun’s course have converged to make these bleak and darkening days for many of the world’s developed nations, and certainly for America. What we need is expert and effective guidance on the impact of policies and programs. What we get is a cacophony of conflicting, often incoherent, ill-informed just-so stories backed by some combination of intuition, self-interest, resentment, herd thinking, natural and social scientific theory, and cherry-picked statistics. The modern social sciences in particular, which had as their mandate and their promise to guide us in times like these, have often become simply another part of the problem, providing dueling experts for hire with dubious track records. That is, when they are not busy generating results that are completely irrelevant to real life practical problems.

How has this happened? We can blame human nature or ideological corruption, but I think it’s time to come to terms with the fact that one of the central activities of social science is a fool’s errand, because a core assumption that underlies it is wrong. That central activity is to create mathematical models that explain social phenomena by identifying and measuring a limited set of contributing causes. This is done using statistical tests for significance, explanatory power, accuracy and reliability (the p-values, F-tests, confidence intervals, factor analyses and so on). With minor variations, this is what the “scientific” work of political science, sociology, educational theory and social psychology consists in. Doing this is what it takes to get published in major journals and achieve tenure at major universities. Even economics uses these and related statistical methods, when it stops being social metaphysics and decides to get dirty with evidence. A core assumption that underlies this work is that there are unchanging relationships between the variables that can be identified in causal models.

Despite millions of hours of effort, the inconvenient truth is that there is not a single non-controversial quantitative model in the social sciences. I don’t mean a qualitative model which reformulates a truism, or is logically derived from prior assumptions. Nor am I referring to a mere statistical snapshot with no claim to durability (though the vast numbers of these too are contested). I mean a robust causal model, with dependent and independent variables, with measured and fixed coefficients giving the relative influence of the independent variables on the result, and applicable beyond the test scenario to a wider range of cases which have been successfully applied with precision. The sort of thing that litters the natural sciences like bones on a particularly grisly battlefield, allowing experts to build hydroelectric dams, synthetic organisms and Xbox 360s by exploiting precise and unchanging mathematical relationships. If there is such a quantitative model (or, one hardly dare utter the word, “law”) in the social sciences, I have not seen it. I’m willing to bet that neither have you.

That’s an embarrassment for the obvious reason that social scientists want to know, at a high level, how to explain, predict and shape human events. Since it may seem that I am overstating the case, let me take a moment to agree that we are often able to explain, predict and shape human events without the use of social science. We are not social idiots colliding into each other blindly. But we do need help, lots of it, to understand the effects of interventions large and small. In understanding these interventions, I’m also happy to agree that statistical correlations can help us to see new patterns and sometimes avoid believing stupid things. So the question is really this: given all that we can do without unchanging quantitative causal models of human behavior, is the quest to find such models largely an empty exercise? Or to express it with more exasperation, is it some combination of fraud and farce?

It isn’t as though similar embarrassing thoughts have never occurred to anyone before.But the many dozens of criticisms of the methodologies have had all-too-little impact on the institutions of social science and the organizations that apply social research. The faith in economic theories of efficient markets and models of asset pricing that directly led to the financial market collapse and global recession is just the most painful recent case in point.

The problem is not just the complexity of social phenomena, or bias in research. If complexity alone were the problem, we could dig down to the bedrock and identify the fundamental laws or model parameters, even if we couldn’t reliably predict the behavior of large systems. In fact, it’s just this faith in a discoverable bedrock that sustains research programs. But there is no such bedrock in the social sciences. Everything we measure shifts. The reason no mathematical constants have been found in the social sciences is not that it is harder to find them, but that there aren’t any to be found.Human beings are neither rational agents in the sense of homo economicus, nor fully irrational or automatic agents perpetually stuck in defined response patterns. We are mildly rational and reasons-responsive, if not at every moment, then when we are alerted or primed to pay attention to our interests and what may or may not achieve them (and no, I have no mathematical model for that).

Being mildly rational implies that if someone has observed a stable pattern of behavior and tries to exploit it to take advantage of it, we, either individually or collectively, have the ability to stop exhibiting the pattern that is being exploited, so long as we are responsive to our interests. That’s a deeper point than it may at first appear. Because any generalization of social science which could be treated as a fixed quantitative relationship to be used for prediction and control, thereby exploiting the individuals concerned, runs into this problem. Rational agents, once aware of the attempt to use their own behaviors against their perceived best interests, or aware that by changing their behavior they can improve their situation, will change their behavior or cease to be rational, because by continuing to act the same way, they are knowingly acting against their interests. Thus, rational agents must break whatever modeled relationship was being exploited.

This insight emerged from Lucas’s critique of economic policy, but even stripped of his strong assumptions about human rationality it retains much of its force. Economic cases are the most obvious: if a hedge fund discovers a regularity in the market and seeks to exploit it, it may succeed for a while in doing so, but as others catch on, they change their trading patterns and the game is soon up. IQ tests, no matter how well-designed to reveal innate, unchangeable intelligence by measuring behavioral responses, can be gamed. The point holds for any limited set of operationalized variables in a social or behavioral model of action. If you don't agree, try to find a counterexample and you’ll get a sense of the futility. If you think you’ve got one, by all means share it in the comments.

There is wisdom here that has even been formulated in “laws,” which are promptly ignored:

Campbell’s law: The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

Goodheart’s law: Any observed statistical [social] regularity will tend to collapse once pressure is placed upon it for control purposes.

So what is the lesson? A good rule of thumb is that we are slippery subjects for a model-building science to the extent we are rational or even quasi-rational, and we are tractable to science to the extent that we are stuck in our ways despite wanting to change. Of course, none of this is to say that correlations are useless, or that qualitative causal models can’t help people think through the implications of their commitments better than untutored intuition. But the impact of variability combined with complexity should be made a more explicit part of social scientific work. At a minimum, every theorist engaged in the statistical model building enterprise should ask whether common knowledge of the measured relationship between variables would tend to strengthen it, break it, or neither. They should then ask whether employing the model as part of an economic, educational or other intervention will tend to sustain or break it. The presumption should be that using it will tend to break it, unless there is strong evidence to the contrary. Having asked themselves, authors should make the answers part of the published paper. Conscientiously applying these principles will not solve all the problems that the social sciences have in advising policy makers, but it would be a start.