4b. Classical Test Theory Is a Triple Scientific Disaster

Classical Test Theory (CTT) underpins most of psychological test validation studies but does not explain how it is possible that a test score would measure a quantity. CTT interprets merely any score as the result of a random experiment. This random experiment is modelized by a probability law, which assigns a probabilty to any possible score when the person is tested. To supply the set of the possible scores with a probability law allows one to define a numerical random variable with a mathematical expectation. This expectation is qualified as the “true score”, in such a way that thanks to this mathematical construction, any score contains a true score and a measurement error, the arithmetical residual that bridges the gap between the two terms (the test score and the true score).

If test score users consider that observed scores contain true scores, and as soon as they adopt a realist interpretation of the truth concept (what is true is what occurs in the real world), they can deduce that these scores measure something. But CTT prevents them to claim that elementary interpretations of test scores are logically valid, because of the unrestricted nature of measurement error. For example, one cannot deduce from the observation, Paul’s anxiety score today is smaller than his anxiety score yesterday, that his true score decreased. Or, Paul’s IQ is higher than Julie’s IQ; this observation does not imply that Paul’s true IQ is higher than Julie’s true IQ. Consequently, the use of CTT as a statistical framework for test validation consists in adopting the epistemological norm according to which invalid interpretations of test scores are acceptable for an assessment practice that proclaims scientificity–psychological tests are deemed to have been scientifically validated.

In the perspective of psychological science, the use of CTT is a triple disaster. Firstly, by obscuring the logical distinction between a valid argument and an invalid argument, it prevents psychologists to have a clear sense of what a scientifically grounded conclusion is. Let us consider an argument of form “if premise then conclusion”: the conclusion is valid if it logically follows from its premise; if the premise is true, then the conclusion is valid and true. For a conclusion, based on a theory and some observations, be scientifically grounded, the observations have to be objective, that is, observer-independent (who can prove that the observations are true?), and the theory has to be corroborated, that is, successfully tested (i.e., tested and not falsified).

Secondly, the use of CTT allows psychologists to ignore that measuring a quantity by observing some phenomena requires–necessary condition–that these phenomena are simply ordered. The truth of the preceding proposition can be tested (see Scoring vs. Measurement), and it is unplausible (see the Definition).

Thirdly, by establishing that numerical aggregation of qualitative facts is the normal approach to objective description in scientific psychology, CTT encourages psychological researchers to underestimate the theoretical stakes of any descriptive methdology, especially by institutionalizing the ignorance of the conceptual difference between simple and partial ordering–which, to my knowledge, is detailed in only one textbook on psychological methodology: “An introduction to the logic of psychological measurement” by Joel Michell (1990).

The present note elaborates on the preceding points by (1) detailing the true score concept, (2) deducing that the comparison of two scores viewed as measurements of true scores confines to divination, (3) highlighting that probabilistic modelization has nothing to do with theorising about measurement functions, and (4) arguing that test scores should not mask the empirical m-tuples on which they are built.

1. The true score

The concept of the true score that is contained in a test score rests on two concepts, namely a scale of possible scores, and a probability function, which assigns a probability to any possible score. To my knowledge, CTT, as any psychometric theory, does not question the existence of probabilities nor does it raise the epistemological issue of how these probabilities could be known in a testable way.

Let us consider a scale of k scores xi, i = 1, 2, …, k. CTT postulates the probability law {pi, i = 1, 2, …, k} (which entails that the sum of the probabilities is equal to unity). It postulates also a random variable the expectation of which is, by definition, the sum of the scale scores weighted by their probabilities, that is,

true score = p1 x1 + p2 x2 + … + pk xk = Σ pi xi.

2. Scores comparison is divination

The insurmountable difficulties that result from this approach are cristal clear: how to deduce anything about the true scores of two scores? For example, Paul and Julie obtained IQ scores of 120 and 130, respectively. Can one deduce that Paul’s true IQ is lower than Julie’s true score? As one ignores the probability laws that are postulated to interpret the two IQs as the result of two random experiments, one ignores the two true scores. The score viewed by CTT is to the psychologist’s desire what Saint-Exupéry’s box is to Petit Prince’s desire: a box that contains an inaccessible sheep, a statistical shell for an inaccessible number.

3. The true score measures nothing

The idealization that yields the postulation of probability laws proliferating in a metaphysical space masks the fundamental problem of measurement. Measurement rests on a metaphysics of quantity, which does not coincide with that of probabilities. Indeed, a probability is a quantity that cannot be measured.

If a probability were measurable, one would be able to invoke an increasing function, which assigns one and only one point in a set of observable values to any point from [0, 1]. I do not see what such a set could be, and it is clear that the set of frequencies (or proportions) is not suitable, since any probability from ]0, 1[ is associated with at least two observable frequencies. It suffices to consider at least two random experiments based on the same probability law, such that the results are distincts. For example, I toss up a coin and I observe heads. I toss up the same coin and I observe tails. The proportion of heads that is associated with probability p is 1 if I consider the series “heads”, and 1/2 if I consider the series “heads, heads”.

CTT obscures the concept of measurement, since it diverts attention from the measurement process that should yield the score via a measurement function, by focusing it on the true score, the value of which remains inaccessible, even by upper and lower bounding.

4. The scores are qualitative compound events

As a statistical theory, CTT occults the empirical, experimental, or instrumental meaning of the scores. Test scores are treated as data in the proper sense of the term. The psychometrician does not wish to know about their origin, but accepts them without reservations, letting to the (non-statistician) psychologist the responsibility of their making. Such a division of labour enables a fools’ game, which aims at precluding the rejection of test scores as unsuitable scientific data.

The note Scoring vs. Measurement details the making process of test scores with the help of descriptive and scoring functions. The former, which is denoted by x, assigns a m-tuple of responses to any observation unit; the latter, a, assigns a score to any m-tuple. No score if no m-tuple. By its confinement to statistical analysis of scores, CTT enables the entire community of score users not to stumble over the evidence that, in order to measure a quantity by using some observables, these observables have to be simply ordered. But psychotechnical observables are m-tuples.

How to simply order m-tuples? For example, how to order (0, 0, 1) and (1, 1, 0)? Or, how to simply order points floatting in a cube? As soon as the descriptive approach is multivariate, its descriptive space is not a simple order because a simple order is a one-dimension space, within which objects are ordinally scaled, that is, posited one after the other with possible ties.

The classical test theoretician omits to notice that “0 and 0 and 1” does not equate with “0 plus 0 plus 1”. Such omission is faulty, since probability theory, which falls in the psychometrics’ area of responsibilty, rests on the concept of event algebra. The logical conjunction “and” is defined as a product instead of a sum, which corresponds to the logical disjunction “or”.  The 3-tuple (0, 0, 1), and more generally a m-tuple is not a sum but a logical product. This is a compound event, a co-occurrence.

Conclusion. It is time that psychologists, in their efforts towards scientificity, retake control of the theoretical elaboration of their data.

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *