Psychological Science: The [Non-]Theory of Psychological Testing - Part 1

This is the first in a planned series of articles with the frontispiece title, "Psychological Science:". To give you an idea of other topics that may develop, here are a few working titles. "Sigmund Freud, a Personal and Scientific Coward", "Classical Inference, Bad. Bayesian Inference, Good", and "Fighting Over Combat Related PTSD". These will not follow in succession, necessarily. Rather, I will intersperse them with other articles under the lead title of, "My Life As a Crime Fighter:", at least one more on "My Father: A Veteran's Story", and other creations as the Muses will inspire. Finally, so that the question doesn't have to be asked, I want to establish my blog creds so I might have a shot at a nomination for next year's "The Quark", a prize for science writing awarded by 3QuarksDaily.com.. My goal is to be nominated by someone other than myself.

In summary

Psychological Test Theory is no such thing. It is a tautology, not a theory. *

* For those who are interested, the remainder of this article is elaboration.

Modern Psychological Test Theory

    Modern Psychological Test Theory (PTT) comes in two flavors: Classical Test Theory (CTT), and Item Response Theory (IRT). They are not competing views of psychological assessment, rather, they are complementary. CTT deals with the total test score, and IRT focuses on the individual items that make up the total test score. CTT looks at your total Algebra test score of 81. IRT studies the response choices for the item. For example,

        Given the equation, x + 4 = 7, solve for x.

                a) x = 11

                b) x = 3

                c) x = 0

                d) x = 1

CTT focuses on average test scores and how they vary across people and groups. IRT wants to know about item difficulty, item discrimination, and probability of guessing. Graduate students in psychological research and psychometrics regard CTT as old hat, and IRT as really cool stuff. Educational Testing Service (ETS) of Princeton, NJ makes millions on both, but the real cash cow is IRT.

    CTT and IRT have two things in common. These things in common are also their two scientific Achilles heals. 1. They both assume a fundamental reality that can never be known completely. For CTT, the fundamental reality is the True Score. For IRT, the fundamental reality is the Latent Trait. These realities are expressed as stand-in concepts that can only approximate the reality. All of CTT is posited on the notion that your actual score on your Algebra final exam is composed of a True Score and Error (as in, "To err[or] is human."). Put succinctly:

        Your Actual Score = Your True Score + Error

This shows that CTT is interested in you, as a person, and not just as a number. And, yes, it is quaint. IRT is based on the principle that an individual test item taps into a Latent Trait, and does so in some amount. The trait of interest might be a personality disorder, like Borderline Personality Disorder. IRT doesn't care a damn about you as a person no matter how over the border your personality is disordered. All it cares about is how your response to an item is used in plotting an Item Characteristic Curve. And, no, it is not quaint. I am going to blame this all on Plato, and maybe Socrates. See below.

    2. The second commonality and second Achilles heal (as if one wasn't enough) is that CTT and IRT are both developed from mathematics and assumptions that are not derived from observations in nature. Don't believe me? Then do what I did in graduate school. Take a course in  PTT using Frederick Lord's and Melvin Novick's (1968) classic textbook, with four chapters on IRT written by Allan Birnbaum, "Statistical Theories of Mental Tests." Make sure you brush up on your calculus. A course on mathematical statistics couldn't hurt, either. If someone proves me wrong, I'll treat the first to notify me to dinner. I'm going to blame this all on Pythagoras. See below.

The problem started with Plato, and maybe Socrates

    Do you see what just happened? We have just stepped into Plato's (c. 428 BCE - c. 348 BCE), and maybe Socrates' (c. 469 BCE - c. 399 BCE), World of Ideal Forms, or World of Ideas. Is this bad? You bet it is. From a scientific point of view, PTT stepped in a big pile of dodo, two piles to be exact.

    Let's look at Plato's Forms or Ideas. Plato proposes two realities: that of the senses or experiences which are subject to change; and that of the unchanging essences in the World of Ideal Forms. In the world of our experiences, we encounter kitchen tables, coffee tables, wooden tables, tables constructed from FedEx boxes, and so many other tables. Each of these expressions of a table in the world of experience, actually partakes of the reality of Tableness from the World of Ideal Forms. For Plato this was not a metaphor. It was as real as real can be. The World of Ideal Forms was so real that it was the ultimate reality, or at least the home of many ultimate realities. A draftsman's table did not so much partake of the Ideal Form of Tableness, as the Ideal Form of Tableness infused the world of experience. Another way of saying this is that the Ideal Form of Tableness acted upon the world of our senses. Tableness was real and it had power.

    Ideal Forms were not limited to those that could yield tangible expressions (like tables) in our sensory world. The World of Ideal Forms was populated, also, with immutable principles and attributes like Justice and Beauty and Intelligence and Depression. These principles acted upon the world of experience and were expressed in our things, and our institutions. The State, properly organized, was the highest expression of Justice in the world of experiences. Thomas Aquinas, proved the existence of the soul and the afterlife with the argument from Justice. In short, since we see an incomplete expression of Justice in our daily lives, and since Justice is real and acts upon God's creation, then scores must be settled and ledgers must be balanced somewhere other than in the world we experience. Aha! Therefore, there is an afterlife. Aha!, again. The afterlife must be populated by souls so that Divine Justice can be visited upon something that outlasted decayed flesh. British humor captured this nicely with an Anglo-Saxon St. Peter. "English souls up this way. The French, down there, please."

Pythagoras was the other culprit

    Pythagoras (c. 569 BCE - c. 475 BCE), initially, had the markings of a budding scientist. He was keen to observe and experiment, at least so far as the physics of sound and music was concerned. His greatest achievement was discovering properties of mathematics and using them to represent phenomenon in the world. A lot of his applications were in geometry. Why geometry? Because geometry was integral to civilized life: construction, land surveying, measuring of objects and commodities, and so on. Then he fell off his goat, damaged his brain, and took a scientific turn for the worse. The hell with wasting time observing, experimenting, and recording data. All of the secrets of the universe would yield, he preached, to the study of the mathematics that underly all of nature. Experience only gets in the way and wastes time. Get into the numbers, the ratios in particular, and all will be revealed to you. To make matters worse, knowledge was to be kept secret among the few, and not promulgated to the masses. The path to knowledge (enlightenment, if you will) was this: create a philosophers' club, require secret passwords and handshakes, support yourself from student tuitions, and delude yourselves into thinking that truth is yours for the thinking.

    Pretty stupid, huh? Well, not so stupid that most of western civilization, for 2,000 years and more, bought into the ideas that thinking and logic obviated any need for observation and experiment. It was also adopted by the church and still influences theology, doctrines of faith, and papal encyclicals. It's also part of the philosophical foundation for PTT. The real scientists-in-the-making like Thales (c. 624 BCE - c. 547 BCE), Democritus (c. 460 BCE - c. 370 BCE), Anaximander (c. 610 BCE - c. 546 BCE), and Anaximenes of Miletus (c. 585 BCE - c. 525 BCE) lost the game. The final score:

        Real Men From Let's Just Think About It            -     8

        Wusses From Let's Observe And Experiment     -     1

Abelard, and Galileo to the rescue

    So, now PTT is up at bat with two strikes against it. This would be warning enough for most good scientists. The problem is that it's impossible to see the forest for the trees if you believe you are using the correct form of good science. Also, it's hard to detect a problem when it appears that PTT does have a great deal of utility and value. It is my personal view that most psychometricians have not the foggiest notion of the consequences of having Plato and Pythagoras as their research associates. They need to fire P&P and hire a new research team, Peter Abelard (1079 CE - 1142 CE), and Galileo Galilei (1564 CE - 1642 CE). Abelard will handle the philosophical foundations, and Galileo will address the scientific foundations. First, a warning - no, two warnings. If you don't manage Abelard carefully, and keep him focused on the problem, he's going to monopolize your time and tell you about all his calamities. Like the time he was neutered by the uncle of his teenage student and lover, Héloise, at the behest of her disapproving father. Galileo can be a self-serving SOB. He's as likely to take your inventions and pawn them off as his own. Don't get me started on the telescope thing. If you keep him in line, and watch him like a hawk, he will take you to the stars.

    Abelard, following his mutilation, and not having anything better to do, returned to a monastic life. He was the greatest logician of his time; he was an outstanding teacher; and he made contributions to philosophy, theology, and ethics that influenced the entire Western world. His presence in France led directly to the founding of the University of Paris. With credentials like that, it's an easy conclusion to draw that he was considered, eventually, a heretic by the Church. Of particular note for this discussion is that he completely debunked Plato's reality of Ideal Forms and replaced it with something modern man and science could sink their teeth into. Abstractions (ideas, thoughts, concepts) do not exist. They have no separate reality. There is only one reality, and that is what we experience. It is the real pain from stubbing my real toe on the real piano leg, and letting out a real vocalization that is laced with loud expletives. From experiences of reality we create abstractions that help us name things, classify them, think about them, and relate them. They help us think and generate more abstractions that help us understand. His philosophical contribution to science was this: An idea has no separate existence of it's own apart from the human mind's ability to conceive it, communicate it, and use it to advance thought, language and knowledge. The implication for science is profound. Knowledge proceeds from both experiencing reality AND thinking about it. The scholastics are really going to be pissed.

    Galileo picked up mathematics where Pythagoras wimped out. For Galileo, mathematics was the language of nature, a point on which Pythagoras would agree. Galileo went further to say that mathematics should find and express relationships in that which we observe in the real world. It was ridiculous for science to meditate on numbers, and propose ratios that described all of nature, without testing to see if nature will go along. It would be even more efficient to bypass the whole dream sequence concoction, and find the ratios and relationships in nature itself. To add insult to injury, Galileo abolished all notions of absolutes in nature. He is the first discoverer of relativity. In short, mass, position, and velocity could never be determined without an arbitrary reference. There is no absolute standard for measuring nature. Aristotle, a virtual canonized doctor of the Church, is discarded in one fell swoop. Oh boy! Do you know what this means? This means that the sole arbiter of knowledge about nature and man is science. Wait till the arbiters of knowledge-through-thinking-alone get a hold of this.

PTT is saved by P. W. Bridgeman, or is it?

    In the excellent TV series on the WWII invasion of Europe by the Allied Forces, "Band of Brothers", an episode or two are devoted to the Battle of the Bulge. In particular, it was about the Battle of Bastogne and the 101st Paratroop Infantry Division of the U.S. Army, "The Screaming Eagles". The 101st was completely surrounded by the German Army, cut off from resupply and reinforcements, exposed in the worst winter in fifty years, out-equipped, and out-gunned by the enemy. The siege of Bastogne was first broken by the tanks of General George S. Patton's Third Army. In the popular mind, Patton was instrumental in rescuing the 101st Division. To this day, no member of the 101st ever acknowledged they needed to be rescued.

    The mathematical statisticians, psychometricians, and practitioners behind the development of PTT have something in common with "The Battered Bastards of Bastogne". Like "The Screaming Eagles", they don't think they need to be rescued, either. It's my personal view, that almost all would acknowledge the important advances in philosophy and science made by Abelard and Galileo. They would not, however, see the problems with PTT that are founded in Plato and Pythagoras, and exposed by Abelard and Galileo. Are they dense, stupid, incorrigible, phony, or deficient in someway? No, not really. Why? Because they have an all purpose trump card, or at least they think they do, that will get them out jail, to the head of the line, past the bouncer at Paris Hilton's club, and in a worst case scenario, a do over. Their magic wand? It's found in modern physics. In particular they are the ideas of P. W. Bridgeman in his 1927 book, "The Logic of Modern Physics".

    The relevant ideas from Bridgeman are the hypothetical construct and operationalism. They are important concepts that should be adopted by psychological science. But they can't fix the problems created by Plato and Pythagoras. I will get into this and other issues for PTT in Part 2 of this article on Monday July 20, 2009. Here's something to ponder in the meantime. The Normal Curve (the Bell Curve) is not derived from nature, does not reflect anything in nature, and does not represent anything in nature. The Norman Curve is an invention of Euler and Gauss and is a purely mathematical construction. It has nothing to do with reality as far as PTT goes. Please take the time to comment on this article, and I hope you will be back for Part 2 on July 20, 2009.

    IF YOU ARE LEARY ABOUT USING YOUR REAL NAME AND EMAIL WHEN YOU COMMENT, USE A PSEUDONYM AND A PROPERLY FORMED EMAIL ADDRESS LIKE, me@my.com. WHO'S TO KNOW?

Happy Father's Day.