Validity as an argument

Module: Validity
Tranche: core (a first course)
Before this: Constructs and construct maps · Classical test theory and reliability
Builds toward: Validity as causation: Borsboom’s view · Fairness in testing: bias, impact, and measurement invariance · Differential item functioning · Gathering validity evidence
Starts threads: A criterion built from the same items inflates validity evidence (→ Gathering validity evidence)
Picks up threads: What we want from a measure: insensitive to nonfocal attributes, calibrated, precise quickly (from What is measurement?)
IRW tables: silk_2019_inattentive, silk_2019_hyperactive
From EDUC 252: slides c3
Status: draft
Authorship: Written largely by Claude (Anthropic), from Ben Domingue’s EDUC 252 materials and under Ben’s direction. Ben reviews each lesson before it is marked done.
Comments: open an issue on GitHub about this lesson.

What this is for

A 2022 study of a rating scale for ADHD-like behaviour in dogs reports that its six inattention items have an alpha of 0.81 (Csibra, Bunford & Gácsi, 2022, sec. 3.2.1). Classical test theory and reliability told us what that number means. It doesn’t say that the scale measures inattention, or that it should be used to decide anything about a particular dog. Those are claims about what scores mean and what they are for, the business of validity. This lesson treats validity as an argument: a chain of inferences from responses to a decision, each needing its own evidence. In the worked example, a symptom count separates children with and without ADHD almost perfectly, and we ask what that separation is evidence for.

Goals

By the end of this lesson you should be able to:

  • say what is validated: an interpretation of scores for a use;
  • lay out an interpretation/use argument (scoring, generalization, extrapolation, decision) and name the evidence each link needs;
  • say what a reliability coefficient licenses and what it doesn’t;
  • recognize a criterion that borrows from the test, and say what agreement with it backs;
  • explain why a validity argument is never finished.

Core ideas

What gets validated?

The Standards for Educational and Psychological Testing define validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” (AERA, APA & NCME, 2014, p. 11). The wording is the point: what is supported is an interpretation, for a use.

Why does the use matter? Take a hypothetical state that gives the SAT to every junior. For admissions, scores are read as readiness for college work, and the evidence that matters is whether they predict how admitted students do. To rank the state’s schools, the same scores are averaged and read as a measure of what schools teach; now what matters is whether a school’s average reflects the school rather than whom it enrols. Evidence for the first use says little about the second.

Recall from What is measurement?: the first item on the list of what we want from a measure was insensitivity to attributes other than the target, so that a mathematics test isn’t also a test of reading English.

Where does that desideratum live? In the argument for a use. The Standards call the threat construct-irrelevant variance, with the same example of a mathematics test that leans on reading (AERA, APA & NCME, 2014, pp. 12–13). Insensitivity to reading is one claim the argument for using the scores with students learning English has to back, and Differential item functioning tests it item by item.

Is validity, then, a property of the test or of what we do with its scores? The answer rests on an assumption. If the attribute exists and differences in it produce differences in responses, validity can be read as a fact about the test, whatever the scores are used for; Borsboom, Mellenbergh and van Heerden (2004) develop that view, and Validity as causation takes it up. If instead what we can check is always a claim made with the scores, for particular respondents and a particular purpose, then evidence for one use need not carry over to another. The Standards take the second view, and advise against speaking of a test’s validity without saying which interpretation and use are meant (p. 11).

Whichever view you hold about what validity is, calling a measure “validated” leaves out which interpretation, for which use and which respondents, the evidence supports. The label travels further than the evidence: a scale studied in one population for one purpose is soon used in another, and “validated” doesn’t say whether the evidence came along.

From types of validity to construct validity

Recall from Constructs and construct maps: a construct is an attribute we postulate, given meaning by Cronbach and Meehl’s sentence linking the attribute, a situation and a probable response. Because we propose it, we could be wrong about it.

The idea that a test “has” a validity is older. The American Psychological Association’s committee on psychological tests (1950–54) set out four types of validity, each established by its own kind of study: predictive, concurrent, content and construct validity (Cronbach & Meehl, 1955, p. 281). Construct validity, the committee’s new term, was needed whenever no criterion and no body of content was accepted as fully defining the attribute (p. 282).

Two points in their paper eroded the separation. A criterion is itself a measure, and sometimes no more valid than the test it checks (p. 284). And a construct makes predictions, so a sound finding against them counts against construct and test together (p. 299): construct validity demands evidence that could have come out the other way. Messick (1995) made construct validity the whole of validity, with content, criterion relations and consequences as aspects of it. The Standards follow him, listing sources of validity evidence rather than types of validity (AERA, APA & NCME, 2014, pp. 13–14). The five sources are test content, response processes, internal structure, relations to other variables, and consequences of testing.

The first three are internal evidence, about the test and its responses, and much of this course; the last two are external, about how scores line up with the rest of what we know about respondents. How do we decide which evidence a particular use needs?

The argument

How do we get from responses to a decision? Kane (1992) proposed treating the route as an argument, and Kane (2013) calls it the interpretation/use argument. It is a chain of inferences, each resting on an assumption (its warrant) that evidence (its backing) must support. Four links are usual (Cook, Brydges, Ginsburg & Hatala, 2015):

  1. Scoring: from responses to a score. Are the items keyed correctly, and do respondents engage with them as intended?
  2. Generalization: from this score to the score over other occasions, forms or raters.
  3. Extrapolation: from the test to the attribute, or performance in the world, that we care about.
  4. Decision: from the score to an action (Cook et al. call this link implications). Does acting on the score do what it should, at acceptable cost?

The argument is laid out before data are collected, for triage: evidence should go to the assumptions most in doubt for this use (Cook et al., 2015), and more ambitious claims need more backing (Kane, 2013). The sources of evidence attach to the links: content and response processes mostly back scoring, internal structure (including reliability) backs generalization, relations to other variables back extrapolation, and consequences back the decision. Cook and Beckman (2006, Table 2) give examples of each source for four kinds of instrument.

Try it: build an argument

Pick a use. For each link, choose the source of evidence you would gather to back it.

The scoring and generalization claims read much the same across uses; the extrapolation and decision claims are where the two SAT uses part company. The mapping is a guide, not a rule: content evidence also backs extrapolation when the claim is that the items represent a domain, as with the blueprints in Constructs and construct maps.

Three questions are worth asking of any validity claim. What is the proposed interpretation and use? What evidence supports it? Is that evidence enough for this use?

Reliability is necessary, not sufficient

Recall from Classical test theory and reliability: reliability is the share of observed-score variance that is true-score variance, and alpha estimates it from one administration, as a lower bound when the item errors are uncorrelated. A test’s correlation with anything is bounded by the square root of its reliability.

Where does reliability sit in the argument? At the generalization link. The bound in the recall makes it necessary: scores that don’t agree with themselves can’t agree well with anything else. Cook and Beckman (2006) give a clinical example. Three blood-pressure readings on a stable patient, within three minutes, come out at 185/100, 80/40 and 140/70. We wouldn’t average them to 135/70 and move on. But is agreement among readings enough?

Try it: three readings

A patient’s true systolic pressure is 135. Each reading adds error with the SD you choose, plus a fixed bias from a miscalibrated cuff.

With the error SD at 30, three readings leave the average uncertain by about ±35. Now set the error SD to 3 and the bias to 20: the readings agree closely, and every one is about 20 too high. Reliability measures agreement among readings, and a consistent bias is invisible to it; only a comparison with something outside the cuff, such as a calibrated device, finds it. That comparison is extrapolation evidence.

Back to the dog scale. Under the classical assumptions, an alpha of 0.81 says that at least about 81% of the variance in these owners’ subscale totals is true-score variance: the six ratings hang together. That backs generalization across items, for one rater. It doesn’t say the items measure inattention rather than, say, obedience, or that another rater would agree. Csibra et al. checked the second, comparing owners’ ratings with dog trainers’, and describe the agreement as fair for inattention and moderate for hyperactivity/impulsivity.

Try it: a criterion that borrows from the test

Agreement with a criterion is the usual backing for extrapolation. What if the criterion already contains the test? Here 1,500 simulated children answer a nine-item screener. An independent criterion (say, impairment judged by a clinician who never sees the screener) flags 15% of them. A “diagnosis” mixes that criterion’s information with the screener’s own count, putting weight w on the count. The Standards list this situation, criterion contamination, among the artifacts a study of test–criterion relations should look for (AERA, APA & NCME, 2014, Standard 11.8, p. 180).

Separation is measured by the AUC: the chance that a randomly chosen flagged child has a higher count than a randomly chosen unflagged one, ties counting half (0.5 is none, 1 is perfect; Gathering validity evidence develops it).

Take a score and a group label (case or non-case). Pick a cut: everyone at or above it is flagged. Sensitivity is the share of cases flagged; specificity is the share of non-cases not flagged. As the cut moves from the top of the scale to the bottom, both change, and plotting sensitivity against \(1 - \text{specificity}\) traces the ROC curve, from (0, 0) to (1, 1).

The AUC is the area under that curve. It has a direct reading: for a randomly chosen case with score \(S_1\) and a randomly chosen non-case with score \(S_0\), \[\text{AUC} = \Pr(S_1 > S_0) + \tfrac{1}{2}\Pr(S_1 = S_0)\] (Hanley & McNeil, 1982). An AUC of 0.5 is chance (a diagonal ROC curve), and 1 is perfect separation: every case scores above every non-case.

What it doesn’t tell you: which cut to use, since it averages over all of them, or how many flagged respondents are cases, which depends on the base rate (problem 1).

At w = 0 the screener’s AUC is about 0.76. As the weight rises the AUC against the diagnosis climbs, to 0.97 at w = 0.5 and exactly 1 at w = 1, where the diagnosis is a cut on the count and the histograms no longer overlap. The AUC against the independent criterion doesn’t move. The rising number measures how much the diagnosis borrowed, not how well the screener finds the children we care about.

A measure is never finished

Suppose an argument has been backed, link by link. Is the work done? Not for long. Campbell (1979) proposed what is now called Campbell’s law: an indicator used to make social decisions attracts pressure to corrupt it, and tends to distort the very processes it was meant to track. A test that measured what schools teach before it ranked them may measure test preparation after. Uses also shift as needs change, and the evidence has to follow (Kane, 2013). Validation is a continuing activity, not a certificate.

That leaves a tension, which we might call the validity paradox. The argument-based view asks a lot, for good reasons, but a complete argument for every use can be very hard to deliver. California’s parent guide to its state tests lists four uses of the results, the first of which is to help parents and teachers talk about a student’s performance (California Department of Education, 2015, p. 5). Backing the first would take a study of whether score reports start or improve conversations between parents and teachers, a different kind of study from any psychometric analysis (problem 4 asks you to design it). The paradox is one reason to consider a narrower account of validity, which Validity as causation takes up.

Simulate

The code below is the contaminated-criterion widget in R, with 5,000 children. For each w it prints the screener’s AUC against the diagnosis and against the independent criterion, the share the diagnosis flags, and how often the diagnosis agrees with the criterion. It is base R and runs in about a second.

With the seed as written, the AUC against the diagnosis rises from 0.766 at w = 0 to 0.969 at w = 0.5 and 1.000 at w = 1, while the AUC against the independent criterion stays at 0.766 throughout. At w = 1 the diagnosis flags 11.5% rather than 15%, since the count takes only ten values. Now change rho, how strongly the criterion tracks the screener’s construct: what does rho <- 0.9 do at w = 0? A study against the contaminated diagnosis would report auc_diag; which column would you want?

Download this code to run it in your own R session.

With real data

Our tables come from the Children’s Attention Project, a community cohort study in Melbourne, as analysed by Silk et al. (2019). The main example, silk_2019_inattentive (nine DSM-IV inattentive symptoms), shows a near-perfect separation between children with and without ADHD. The contrast, silk_2019_hyperactive (nine hyperactive/impulsive symptoms for the same children), shows where the separation comes from. Download the code; it needs only base R.

How were the data gathered? Grade 1 children at 43 Melbourne schools were screened with parent and teacher rating scales, and children rated high by both were matched, by sex and school, with children rated low by both. An interviewer then took each parent through a structured diagnostic interview, the DISC-IV (Shaffer, Fisher, Lucas, Dulcan & Schwab-Stone, 2000), which records whether each DSM-IV symptom is present and whether the child meets the criteria for a diagnosis. The tables hold 146 children with ADHD who had never taken medication for it and 209 children without ADHD, aged 6 to 8 (Silk et al., 2019, Methods). Items are named by symptom (avoid, fidget); a response is 1 when the parent reported the symptom, so higher means more symptoms. There is one wave and nothing is missing.

The study was built for a network analysis of which symptoms are central and how they combine, not to validate a screener. We are asking the data a different question.

Code 
# The two IRW tables, from the CSV links on their landing pages (pinned to one version).
base <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse_4:v7_0."
ina  <- read.csv(paste0(base, "silk_2019_inattentive/rows?format=csv"))
hyp  <- read.csv(paste0(base, "silk_2019_hyperactive/rows?format=csv"))
head(ina[, c("id", "item", "resp", "cov_ADHD", "cov_age")], 3)
id item resp cov_ADHD cov_age
1 avoid 0 0 6.970568
1 closeatt 0 0 6.970568
1 loses 0 0 6.970568
Code 
c(rows_inattentive = nrow(ina), rows_hyperactive = nrow(hyp),
  children = length(unique(ina$id)), missing = sum(is.na(ina$resp)) + sum(is.na(hyp$resp)))
rows_inattentive rows_hyperactive         children          missing 
            3195             3195              355                0
Code 
# One row per child. Both tables were built from the same rows of the study's data
# file, so `id` matches across them; the check below confirms that the covariates
# agree child by child. resp is 1 when the parent reported the symptom.
wide <- function(d) {
  w <- reshape(d[, c("id", "item", "resp")], idvar = "id", timevar = "item", direction = "wide")
  names(w) <- sub("^resp\\.", "", names(w))
  w[order(w$id), ]
}
wi <- wide(ina); wh <- wide(hyp)
kids <- unique(ina[, c("id", "cov_ADHD", "cov_age", "cov_male")])
kids <- kids[order(kids$id), ]
kids_h <- unique(hyp[, c("id", "cov_ADHD", "cov_age", "cov_male")])
all.equal(kids, kids_h[order(kids_h$id), ], check.attributes = FALSE)
[1] TRUE
Code 
items_i <- setdiff(names(wi), "id"); items_h <- setdiff(names(wh), "id")
kids$inattentive <- rowSums(wi[, items_i])
kids$hyperactive <- rowSums(wh[, items_h])
table(ADHD = kids$cov_ADHD)
ADHD
  0   1 
209 146
Code 
round(range(kids$cov_age), 1)
[1] 6.4 8.7
Code 
# Alpha for each nine-item count (the function from the reliability lesson).
alpha <- function(x) {
  k <- ncol(x)
  k / (k - 1) * (1 - sum(apply(x, 2, var)) / var(rowSums(x)))
}
round(c(inattentive = alpha(wi[, items_i]), hyperactive = alpha(wh[, items_h])), 2)
inattentive hyperactive 
       0.92        0.88

Both counts are quite reliable by the standard of the reading test in the previous lesson (alpha 0.90): 0.92 for the inattentive items and 0.88 for the hyperactive ones. Now the extrapolation question: does the inattentive count pick out the children with ADHD?

Code 
# How well does each count separate the two groups? The AUC is the chance that a
# randomly chosen child with ADHD has a higher count than a randomly chosen child
# without, counting ties as half: 0.5 is no separation, 1 is perfect.
auc <- function(score, group) {
  r <- rank(score); n1 <- sum(group == 1); n0 <- sum(group == 0)
  (sum(r[group == 1]) - n1 * (n1 + 1) / 2) / (n1 * n0)
}
table(inattentive_count = kids$inattentive, ADHD = kids$cov_ADHD)
                 ADHD
inattentive_count   0   1
                0 127   0
                1  34   1
                2  23   3
                3   7   1
                4  13   4
                5   5   6
                6   0  36
                7   0  32
                8   0  37
                9   0  26
Code 
with(kids, c(without_ADHD_6plus = sum(inattentive >= 6 & cov_ADHD == 0),
             with_ADHD_6plus    = sum(inattentive >= 6 & cov_ADHD == 1)))
without_ADHD_6plus    with_ADHD_6plus 
                 0                131
Code 
round(c(inattentive = auc(kids$inattentive, kids$cov_ADHD),
        hyperactive = auc(kids$hyperactive, kids$cov_ADHD)), 2)
inattentive hyperactive 
       0.99        0.93

The inattentive count gives an AUC of 0.99 (the hyperactive count, 0.93). No child without ADHD has more than five inattentive symptoms, and 131 of the 146 with ADHD have six or more. What about the other 15?

Code 
# The DSM-IV symptom rule: six or more of the nine symptoms in either domain.
kids$rule <- kids$inattentive >= 6 | kids$hyperactive >= 6
table(six_or_more_in_either = kids$rule, ADHD = kids$cov_ADHD)
                     ADHD
six_or_more_in_either   0   1
                FALSE 209   0
                TRUE    0 146
Code 
# Which domain(s) meet the rule, among the children with ADHD.
with(kids[kids$cov_ADHD == 1, ],
     table(inattentive_6plus = inattentive >= 6, hyperactive_6plus = hyperactive >= 6))
                 hyperactive_6plus
inattentive_6plus FALSE TRUE
            FALSE     0   15
            TRUE     61   70

Six or more symptoms in either domain matches the ADHD indicator for all 355 children: 209 of 209 without, 146 of 146 with. The children with ADHD split into 61 who meet the rule on inattention only, 15 on hyperactivity only (the 15 above) and 70 on both, the presentations Silk et al. report. This is the DSM-IV symptom rule, applied to the same parent reports. The diagnosis also requires impairment, and Silk et al. left out three children who met the symptom count without it; within these tables, count and diagnosis coincide by design.

What is the AUC of 0.99 evidence for? Not extrapolation. This is the widget at w = 1: the criterion is a rule applied to the items, so the separation had to be near perfect whatever the symptoms mean. It backs the scoring and decision links in a narrow sense: the interview’s rule was applied as written, and the item codes reproduce it. The sample design adds a second reason for clean separation: children were chosen because parents and teachers both rated them high, or both low, so few from the middle are in the study. An extrapolation claim, that the count tracks inattention that impairs a child’s life, needs a criterion the parent’s symptom reports don’t define: a clinician’s judgement made without the interview, or how the children are doing years later. Silk et al. (2019) related their symptom networks to clinical and cognitive outcomes, which aren’t in the IRW tables.

Problems

  1. Base rates. A screener flags 90% of children who have a condition (sensitivity) and clears 90% of those who don’t (specificity); 5% have it. What fraction of flagged children have it, and what does that imply for the decision link of universal screening? (Meehl & Rosen, 1955.)
  2. A different cut. Redo the Silk cross-tabulation with a cut of five inattentive symptoms instead of six. How many children without ADHD are flagged, and how many with ADHD missed? Which link is this evidence about, and would that change if the diagnosis had been made without the parent interview?
  3. Two arguments for one test. Write a four-link argument for the SAT as an admissions test, and another for ranking a state’s schools (both hypothetical). Which link is weakest in each, and what study would you run first?
  4. A study of conversations. Design the study that would back the first use in California’s parent guide, helping parents and teachers talk about a student’s performance: who is sampled, what is compared, what is measured. Would you expect to find it in a test’s technical report? Why or why not?
  5. Screening for reading difficulty. California requires schools to screen children in kindergarten to grade 2 for risk of reading difficulties, and in December 2024 announced four approved screeners (California Department of Education, 2024, news release 24-53); in the interest of disclosure, I have worked with two of the four groups. Running a validity study for one screener, what evidence would you gather first, and for which link? What changes if its results are also used to evaluate teachers?
  6. Challenge: can an argument ever be complete? Given Campbell’s law, can a validity argument for a high-stakes use ever be complete, or only complete for a time? What would you monitor to know when an argument that once held no longer does? This is open.

Stuck, or curious about something this lesson doesn’t cover? Ask Claude about this lesson opens a new conversation that already knows what the lesson is about; add your question at the end. Claude shows a caution notice on any prompt that arrives through a link; that’s expected.

Going further

  • AERA, APA, & NCME. (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.testingstandards.net/open-access-files.html. Chapter 1 defines validity and describes the five sources of evidence; open access.
  • Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957. The four types of validity, and the case for construct validity.
  • Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741. Validity as a unified concept, with six aspects of construct validity, consequences among them.
  • Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000. The interpretation/use argument at length, with eight general points about validation.
  • Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: A practical guide to Kane’s framework. Medical Education, 49(6), 560–575. https://doi.org/10.1111/medu.12678. A readable walk through the four inferences, with examples from medical education.
  • Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119(2), 166.e7–166.e16. https://doi.org/10.1016/j.amjmed.2005.10.036. The five sources of evidence with clinical examples, including the blood-pressure readings.
  • Hubley, A. M., & Zumbo, B. D. (2011). Validity and the consequences of test interpretation and use. Social Indicators Research, 103(2), 219–230. https://doi.org/10.1007/s11205-011-9843-4. On the place of consequences in validity.
  • Markus, K. A., & Borsboom, D. (2013). Frontiers of test validity theory: Measurement, causation, and meaning. Routledge. https://doi.org/10.4324/9780203501207. Validity theory at book length, including the causal view taken up in the next lesson in this module.

Data sources

Data from the Item Response Warehouse (each table name links to its IRW page); references from IRW biblio.

For instructors

  • Source: EDUC 252 class 3 (slides c3, “A reminder about reliability” through “The validity paradox”, slides 2–22) and class 4 (slides c4, 2–3 and the small-group discussion on slides 16–17). The Silk et al. analysis and the contaminated-criterion simulation are new.
  • Timing: about 45 minutes for the core ideas and widgets; the real-data section works as a 20-minute lab, and problems 3 and 5 as small-group discussions (slide 17 asks groups to be specific about how educators will use the screener).
  • Changed from 252: the slides quote the dog study’s alpha sentences and Cook and Beckman’s description of validity; here both are paraphrased, and the lesson’s one quotation is the Standards’ definition. The Cronbach–Meehl sentence is recalled from Constructs and construct maps rather than quoted again. The SAT uses stay hypothetical, as on slide 9. The CAASPP uses are taken from the California Department of Education’s 2015 grade 11 parent guide (an archived copy of the CDE original).
  • Held back: each source of evidence worked through with data, including classification accuracy and the ROC curve (in Gathering validity evidence, which returns to the ADHD table); validity as causation (in Validity as causation); item-level insensitivity to nonfocal attributes (in Differential item functioning); the same scores used to evaluate an intervention and to measure a construct (in Measurement invariance under treatment and life events); cut scores as decisions (in What does a score mean?).
  • Solutions: held back in solutions/validity-argument.qmd (not published). Problem 6 is open.