What does a score mean? Norms, error bands, and cut scores

Module: Using calibrated items
Tranche: core (a first course)
Before this: Classical test theory and reliability · Information, precision, and short forms
Builds toward: none
Picks up threads: One standard error of measurement for everyone (from Classical test theory and reliability); Whether a scale has an equal unit changes group comparisons (from What is measurement?)
IRW tables: c19prc_uk_mcbride_2021_phq9, coroiu_2018_phq9
From EDUC 252: n/a
Status: draft
Authorship: Written largely by Claude (Anthropic), from Ben Domingue’s EDUC 252 materials and under Ben’s direction. Ben reviews each lesson before it is marked done.
Comments: open an issue on GitHub about this lesson.

What this is for

A respondent scores 12 on the PHQ-9, a nine-item depression screener. What does the 12 mean? On its own, not much. It means something once we say whom it is compared with, how far it might move, and which side of a decision it falls on. Classical test theory and reliability gave every score one standard error; Information, precision, and short forms let the error change along the scale. This lesson puts both to work on the numbers respondents are given. The example is the PHQ-9 in a UK panel surveyed in the first week of the 2020 lockdown, with a German sample for contrast.

Goals

By the end of this lesson you should be able to:

  • turn a raw score into a percentile rank, z score, linear T and normalized T, and say what each assumes;
  • put an error band around a score, from one SEM or a conditional SEM, and say when the two differ;
  • compute the reliability of a difference score and a reliable change threshold;
  • treat a cut score as a decision, do an Angoff calculation, and say why classification near a cut is uncertain;
  • say what a score report should carry, and why group statistics use plausible values.

Core ideas

A score needs a reference

A raw score becomes interpretable in one of two ways (Glaser, 1963). A norm-referenced reading compares it with other people’s scores; a criterion-referenced reading compares it with a standard fixed in advance, such as the screener’s cut of 10. The Standards ask that users be told what a reported score means and where its meaning runs out (AERA, APA & NCME, 2014, open access, Standard 5.1). Cut scores come later; first, norms.

The standard norm-referenced scores are (Angoff, 1971; Kolen & Brennan, 2014, ch. 9):

  • the percentile rank: the percentage of the reference group scoring below, plus half of those scoring the same;
  • the z score, \((x - \bar x)/s\), and the linear T score, \(50 + 10z\);
  • the normalized T score, \(50 + 10\,\Phi^{-1}(\text{percentile rank}/100)\), which puts the percentile rank on a normal scale. PROMIS reports its measures as T scores centred on a reference population (Cella et al., 2010).

Recall from What is measurement?: if a score is only ordinal, any order-preserving rescaling of it is as legitimate as the original, and only conclusions that survive every such rescaling are safe.

A percentile rank uses nothing but order, so it survives every rescaling and needs no claim about equal units. A linear T keeps the raw distribution’s shape and spacing; a normalized T imposes a normal shape. On a skewed distribution, such as PHQ-9 scores in the general population, the two part in the tail.

Each is a comparison with one group at one time; the Standards ask that norms describe that group clearly and be renewed often enough to stay accurate (Standards 5.8 and 5.9).

Try it: one score, four reference groups

The PHQ-9 sum scores analysed below: UK adults in March 2020, two UK age bands, and German adults in 2012. The table converts one score; the panels draw each group, with a movable cut.

A 10 has a percentile rank of 79.0 among all UK adults, 64.7 among 18–34-year-olds, 90.1 among those 55 and over, and 96.2 in the German sample: one score, four meanings. At 20, the UK linear T is 73.5 and the normalized T 68.6; the long right tail stretches one and not the other. At 0, 28.5% of the UK sample are tied at a percentile rank of 14.2.

In the panels, the same cut of 10 flags 36.9% of UK 18–34-year-olds, 22.1% of all UK adults, 10.5% of those 55 and over and 4.1% of the German sample. Slide it to 15 and the four shares are 18.5%, 10.9%, 4.6% and 1.2%. A fixed cut flags very different shares of different groups.

Error bands: one SEM, or a conditional one

Recall from Classical test theory and reliability: the standard error of measurement, \(\sigma_E = \sigma_X\sqrt{1 - \rho_{XX'}}\), is the typical distance between a respondent’s observed and true scores, and CTT gives the same one to everyone, whatever their score.

The simplest band is \(x \pm 1.96\,\sigma_E\), the true scores consistent with what we observed (Lord & Novick, 1968). With the PHQ-9’s SD of 6.22 and alpha of 0.92 in the UK sample below, the SEM is 1.74 and the band ±3.4 points around every score: a 12 has a band from 8.6 to 15.4, which straddles the cut.

Recall from Information, precision, and short forms: test information \(I(\theta)\) is the sum of item information, and the conditional SEM, \(1/\sqrt{I(\theta)}\), is small where many informative items sit and grows where they run out. A screening scale spends its precision where it makes its decisions.

The conditional SEM gives bands that change width along the scale, and the Standards ask for the standard error near each cut score (Standard 2.14; Kolen, Zeng & Hanson, 1996, carry conditional SEMs to a reported scale). The PHQ-9’s items have four categories, so the model below is the graded response model (Samejima, 1969), with one slope per item and a boundary between adjacent categories (Models for polytomous responses develops it). All we need here is that its information adds over items, and that it says how much the sum score varies across replications at a given \(\theta\).

If you’ve done Factor analysis II: omega can stand in for alpha in \(\sigma_X\sqrt{1 - \rho}\). It changes the size of the one SEM, not the fact that there is one.

Try it: bands around a PHQ-9 score

From the graded model fitted below. The upper panel is in PHQ-9 score points: for each sum score, the light bar is CTT’s band and the dark bar the model’s, using the spread of the sum score at the \(\theta\) whose expected score is that sum. The lower panel is the conditional SEM on the \(\theta\) scale.

Near the cut the model’s band is wider than CTT’s (7.2 to 16.8 at a score of 12); at a score of 1 it is narrower, because a sum at the floor can’t move far. Yet on the \(\theta\) scale (lower panel) the conditional SEM grows with distance from the middle. Both are right: the expected-score curve flattens near the floor and ceiling, so there one score point covers a wide stretch of \(\theta\), and the error in points divided by the curve’s slope approximates the error in \(\theta\) (shown below). A low score is precise about the score and imprecise about how far below the cut the respondent is.

Difference scores and reliable change

When the question is whether a respondent changed, the score of interest is a difference, \(D = X_2 - X_1\). For two occasions with equal variances, reliabilities \(\rho_1\) and \(\rho_2\) and correlation \(r_{12}\), its reliability is \[\rho_D = \frac{(\rho_1 + \rho_2)/2 - r_{12}}{1 - r_{12}}\] (Lord, 1956). The higher the occasions correlate, the less of the difference is true change. Cronbach and Furby (1970) asked whether change should be measured this way at all; Rogosa and Willett (1983) showed that when respondents differ appreciably in how much their true scores change, the difference can be quite reliable. Which applies depends on the data.

For one respondent, does the change exceed what error alone would produce? The reliable change index of Jacobson and Truax (1991) divides the change by the standard error of a difference, \(\sqrt 2\,\sigma_E\), so a change counts as reliable at the 5% level when \(|x_2 - x_1| \ge 1.96\sqrt{2}\,\sigma_E\). The Standards ask for precision data for such differences (Standard 2.4).

Try it: how reliable is a change?

At the PHQ-9’s values the difference has reliability 0.73, and a change must be 4.9 points to count. Raise the correlation to 0.85 and the reliability falls to 0.47 while the threshold stays put: the threshold depends only on the SEM. Lower the reliability to 0.80 and the threshold grows to 7.7 points, over a quarter of the scale.

Cut scores are decisions

A cut score turns a score into a category. Where does the number come from? There are two routes, and both end in a choice.

The empirical route sets the cut against an outside criterion. The PHQ-9’s 10 was chosen against diagnostic interviews: in its first validation a score of 10 or more had a sensitivity and a specificity of 88% for major depression (Kroenke, Spitzer & Williams, 2001), and an individual-participant meta-analysis found that 10 maximized the two combined (Levis, Benedetti & Thombs, 2019). An earlier meta-analysis, though, found no substantial differences in pooled sensitivity and specificity across cuts from 8 to 11 (Manea, Gilbody & McMillan, 2012). The data narrow the range; choosing within it trades missed cases against false alarms. Accuracy against the criterion belongs to Validity evidence.

The judgmental route asks experts. In the Angoff method (Angoff, 1971; Cizek & Bunch, 2007, ch. 6) each judge pictures a borderline respondent, just good enough to pass, and estimates for every item the probability that this respondent gets it right. A judge’s cut is the sum of their probabilities; the panel’s is the mean over judges, with a standard error from their spread. The bookmark method orders items by difficulty and asks where the borderline respondent’s mastery stops (Cizek & Bunch, 2007, ch. 10).

Try it: an Angoff panel

Ten Rasch items, difficulties −2 to 2. Each judge’s borderline respondent is scattered around the panel’s, and ratings carry a little noise.

At the defaults judge 4 (a cut of 3.41, against 5.48 to 6.10) pulls the panel’s cut, 5.39 (standard error 0.51), down to θ = 0.24, below the intended 0.5. With no disagreement the standard error is 0.07 and the cut lands at θ = 0.50. With disagreement 1 it is 1.01, and 15 judges bring it to 0.50, roughly the square-root law.

Either way, the Standards ask that a cut’s rationale be documented and, where feasible, informed by data on the criterion (Standards 5.21 and 5.23). And a respondent whose true score sits at the cut would screen positive on about half of hypothetical replications; the Go deeper callout turns that into a number for a test.

Assumptions: an item response model holds with known item parameters; responses are locally independent given \(\theta\); two administrations are independent given \(\theta\) (parallel forms, or the same form with no memory of the first); respondents are classified by their sum score \(X\) against a cut \(C\).

Given \(\theta\). Local independence lets us compute the sum score’s distribution at each \(\theta\) exactly, by adding items one at a time (Lord & Wingersky, 1984). Start with \(f_0(0) = 1\); after item \(i\), with category probabilities \(P_{ik}(\theta)\) for scores \(k = 0, 1, \dots\), \[f_i(s) = \sum_k f_{i-1}(s - k)\,P_{ik}(\theta).\] Then \(p(\theta) = \Pr(X \ge C \mid \theta) = \sum_{s \ge C} f_I(s)\), and two independent administrations agree with probability \[\kappa(\theta) = p(\theta)^2 + \big(1 - p(\theta)\big)^2,\] which is near 1 far from the cut and falls to 1/2 where \(p(\theta) = 1/2\).

Over respondents. The test’s classification consistency is the average of \(\kappa(\theta)\) over the ability distribution, \(\int \kappa(\theta)\,g(\theta)\,d\theta\), with \(g\) the fitted distribution, which needs only one administration. Lee (2010) sets out this approach for tests that mix dichotomous and polytomous items. The average is high when most respondents sit far from the cut, whatever happens near it, so a consistency above 0.9 can coexist with near coin-flip classification for the respondents closest to the cut. The Simulate section checks both against two simulated administrations.

Without an item response model. Livingston and Lewis (1995) build the same quantity from classical ingredients: the reliability gives an effective test length, a four-parameter beta distribution stands in for the true scores, and a binomial for the error around them.

What goes in a score report

Reviews of how score reports are designed and read suggest giving the score with its band, the reference group and its date, the cut and what crossing it means, and what the score does not show (Goodman & Hambleton, 2004; Zenisky & Hambleton, 2012; Standard 6.10). A PHQ-9 report would add that the manual treats 10 or more as a flag for a possible clinically significant condition, not a diagnosis (PHQ instruction manual, n.d., p. 2).

If you’ve done Estimating abilities: EAP estimates shrink toward the mean, so averaging them understates differences between groups. That lesson’s simulation makes the point with numbers.

Reports about groups differ: the best estimate for each respondent is not the best input for describing the group. Estimates that borrow from a prior, such as the posterior mean (EAP; Estimating abilities), are pulled toward the centre, most where the test is imprecise, so their spread is smaller than the population’s. Plausible values are draws from each respondent’s posterior instead of its mean; a statistic computed on each set of draws and combined recovers the population’s spread and carries measurement error into its standard error (Mislevy, Beaton, Kaplan & Sheehan, 1992; Wu, 2005). They are for group statistics only.

Simulate

5,000 respondents take a PHQ-like graded test twice, with a cut at 10. How often does a respondent land on the same side both times? Compare the model’s prediction (the Go deeper callout) and CTT’s, which treats the two scores as normal with correlation alpha. Base R; seconds.

With the seed as written, 92.7% are classified the same way twice; the model predicts 93.1% and CTT’s normal approximation 90.4%, because the scores pile up at the floor, where a normal curve puts too few. Among the 10.4% whose chance of screening positive is between 0.2 and 0.8, only 55.6% are classified the same way twice. Try cut <- 5 and cut <- 20.

Download this code to run it in your own R session.

With real data

The main example is c19prc_uk_mcbride_2021_phq9: the PHQ-9 in the COVID-19 Psychological Research Consortium’s UK panel (Bentall et al., 2022). Wave 1 is an online sample of 2,025 adults with quotas on age, sex and household income, surveyed from 23 to 28 March 2020, the first week of lockdown; 1,406 were followed up a month later (McBride et al., 2021). Its job is to supply real norms, a real cut and two waves. The PHQ-9 asks how often in the last two weeks the respondent was bothered by each of nine symptoms, beginning with “Little interest or pleasure in doing things” (PHQ instruction manual, n.d., p. 5; Kroenke, Spitzer & Williams, 2001). Items are scored 0 (not at all) to 3 (nearly every day), none reversed. The code needs mirt (Chalmers, 2012); download it.

Code 
library(mirt)
# Each IRW table has a landing page (itemresponsewarehouse.org/tables/<name>/) with a
# CSV download. These links are pinned to one version of the data.
uk_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse_4:v7_0.c19prc_uk_mcbride_2021_phq9/rows?format=csv"
de_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse_6:v3_4.coroiu_2018_phq9/rows?format=csv"
uk <- read.csv(uk_url)
# Keying: every item is 0 (not at all) to 3 (nearly every day); higher = more
# frequent symptoms, and none is reversed. The panel has six waves; wave 1 is the
# March 2020 baseline and wave 2 the follow-up a month later.
table(uk$wave[!duplicated(paste(uk$id, uk$wave))])     # respondents per wave

   1    2    3    4    5    6 
2025 1406 2019 3867 2520 2058
Code 
long2wide <- function(df) {
  wide <- tapply(df$resp, list(df$id, df$item), function(x) x[1])
  as.data.frame(wide[, order(colnames(wide))])
}
w1 <- long2wide(uk[uk$wave == 1, ])
age <- tapply(uk$cov_age, uk$id, function(x) x[1])[rownames(w1)]
c(respondents = nrow(w1), complete = sum(complete.cases(w1)))
respondents    complete 
       2025        2025

Everyone answered all nine items. A check against Shevlin et al. (2020), who report that 22.1% of this sample scored 10 or more, with alpha 0.92:

Code 
# Check the pipeline against the published figures for wave 1 (Shevlin et al., 2020).
x <- rowSums(w1)
alpha <- function(X) { k <- ncol(X); k / (k - 1) * (1 - sum(apply(X, 2, var)) / var(rowSums(X))) }
skew <- function(v) mean((v - mean(v))^3) / sd(v)^3
round(c(mean = mean(x), sd = sd(x), skew = skew(x), share_zero = mean(x == 0),
        share_10_plus = mean(x >= 10), alpha = alpha(w1)), 3)
         mean            sd          skew    share_zero share_10_plus 
        5.378         6.220         1.291         0.285         0.221 
        alpha 
        0.921

We get 22.1% and 0.921. The mean is 5.38, the SD 6.22 and the skew 1.29; 28.5% score 0.

Norms: what is a 10?

Code 
# Percentile rank: the percentage scoring below, plus half of those scoring exactly
# the same (the midpoint convention). It uses only the order of the scores.
pr <- function(score, ref) 100 * (mean(ref < score) + mean(ref == score) / 2)
band <- cut(age, c(17, 34, 54, Inf), labels = c("18-34", "35-54", "55+"))
groups <- c(list("UK, all" = x), split(x, band))
round(t(sapply(groups, function(g) c(n = length(g), mean = mean(g),
  share_10_plus = mean(g >= 10), pr_of_10 = pr(10, g)))), 2)
           n mean share_10_plus pr_of_10
UK, all 2025 5.38          0.22    78.96
18-34    626 7.82          0.37    64.70
35-54    763 5.05          0.20    81.39
55+      636 3.37          0.11    90.09

A 10 has a percentile rank of 79.0 overall, 64.7 among 18–34-year-olds and 90.1 among those 55 and over. “Unusual for a UK adult in March 2020” and “unusual for your age” are both reasonable norms, and they give different answers.

The contrast is coroiu_2018_phq9, the same items in German, from adults representative by age and sex, interviewed at home in 2012 (Coroiu et al., 2018). Its job is to show how much a norm depends on its reference group.

Code 
de <- read.csv(de_url)
wde <- long2wide(de)
xde <- rowSums(wde[complete.cases(wde), ])    # 43 respondents skipped an item
round(c(n = length(xde), mean = mean(xde), sd = sd(xde),
        share_10_plus = mean(xde >= 10), pr_of_10 = pr(10, xde)), 3)
            n          mean            sd share_10_plus      pr_of_10 
     2404.000         2.290         3.213         0.041        96.215

Of the 2,404 respondents who answered every item, 4.1% score 10 or more, and a 10 has a percentile rank of 96.2. An earlier German household survey found a similar share, 5.6% (Kocalevent, Hinz & Brähler, 2013). Is the UK’s 22% the pandemic’s doing, then? These data can’t say. The samples differ in country, language, year, sampling (an online quota panel against a household sample) and mode (a screen against an interviewer), and any of these can move a symptom count. What the comparison shows is how much a percentile rank depends on its reference group: the same 10 is the 79th percentile in one and the 96th in the other.

Code 
# A linear T keeps the raw distribution's shape; a normalized T forces it to be normal.
linear_T     <- function(score, ref) 50 + 10 * (score - mean(ref)) / sd(ref)
normalized_T <- function(score, ref) 50 + 10 * qnorm(pr(score, ref) / 100)
round(sapply(c(0, 5, 10, 20), function(s)
  c(raw = s, pr = pr(s, x), linear_T = linear_T(s, x), normalized_T = normalized_T(s, x))), 1)
             [,1] [,2] [,3] [,4]
raw           0.0  5.0 10.0 20.0
pr           14.2 61.7 79.0 96.8
linear_T     41.4 49.4 57.4 73.5
normalized_T 39.3 53.0 58.1 68.6

The two T scores are 49.4 and 53.0 at a raw 5, but 73.5 and 68.6 at a raw 20: the skew separates them in the tail.

Error near the cut

Code 
# One SEM for everyone, and a 95% band of 1.96 SEM either side of each score.
sem <- sd(x) * sqrt(1 - alpha(w1))
pos <- x >= 10                                     # screened positive
round(c(sem = sem, half_width = 1.96 * sem,
        positives = sum(pos), band_reaches_cut = mean(x[pos] - 1.96 * sem <= 10)), 3)
             sem       half_width        positives band_reaches_cut 
           1.745            3.420          448.000            0.420

CTT’s band is ±3.42 points. Of the 448 who screened positive, 42.0% have a band reaching down to 10: the data are consistent with a true score under the cut.

Code 
# The graded response model: one slope per item and a boundary between each pair of
# adjacent categories. mirt writes each boundary as a*theta + d_k.
set.seed(61)
grm <- mirt(w1, 1, itemtype = "graded", verbose = FALSE)
round(coef(grm, simplify = TRUE)$items, 2)
        a1    d1    d2    d3
Dep_1 3.24 -0.62 -3.21 -5.61
Dep_2 3.61 -0.28 -3.65 -5.86
Dep_3 2.21  0.05 -2.06 -3.67
Dep_4 2.71  0.08 -2.56 -4.23
Dep_5 2.60 -1.27 -3.09 -4.87
Dep_6 3.39 -1.88 -3.84 -5.95
Dep_7 3.45 -1.48 -3.94 -6.10
Dep_8 3.12 -3.32 -4.71 -6.81
Dep_9 3.36 -3.43 -5.04 -6.94
Code 
# The cut on the theta scale: where the expected PHQ-9 score (the test characteristic
# curve) reaches 10.
tcc <- function(t) expected.test(grm, matrix(t))
theta_cut <- uniroot(function(t) tcc(t) - 10, c(-4, 4))$root
csem <- function(t) 1 / sqrt(testinfo(grm, matrix(t)))
round(c(theta_cut = theta_cut, csem_at_cut = csem(theta_cut),
        csem_at_0 = csem(0), csem_at_minus_1 = csem(-1), csem_at_minus_2 = csem(-2)), 2)
      theta_cut     csem_at_cut       csem_at_0 csem_at_minus_1 csem_at_minus_2 
           0.86            0.21            0.27            0.77            2.80

The slopes run from 2.21 (Dep_3) to 3.61 (Dep_2), all above the 1 to 2.2 where most financial-literacy slopes fell in Information. The cut sits at \(\theta = 0.86\), where the conditional SEM is 0.21; it is 0.27 at \(\theta = 0\), 0.77 at \(-1\) and 2.80 at \(-2\). It is precise where the decision is made and says little about how far below it the 28.5% who scored 0 sit.

Code 
# The same precision on the raw-score scale: the SD of the PHQ-9 sum for respondents
# at a given theta, from the model's category probabilities.
raw_csem <- function(t) {
  v <- sapply(1:9, function(i) {
    p <- probtrace(extract.item(grm, i), matrix(t))
    sum((0:3)^2 * p) - sum((0:3) * p)^2
  })
  sqrt(sum(v))
}
at <- sapply(c(1, 3, 5, 10, 15, 20, 25), function(s) uniroot(function(t) tcc(t) - s, c(-6, 6))$root)
# The two scales are linked by the slope of the test characteristic curve: an error of
# SE points in the score is an error of about SE / slope on theta. Where the curve is
# flat (near the floor and the ceiling), a small band in points is a wide band in theta.
slope <- sapply(at, function(t) (tcc(t + 0.001) - tcc(t - 0.001)) / 0.002)
round(rbind(raw = c(1, 3, 5, 10, 15, 20, 25), theta = at,
            model_sem_raw = sapply(at, raw_csem), ctt_sem_raw = sem,
            tcc_slope = slope, sem_raw_over_slope = sapply(at, raw_csem) / slope,
            csem_theta = sapply(at, csem)), 2)
                    [,1] [,2] [,3]  [,4]  [,5]  [,6]  [,7]
raw                 1.00 3.00 5.00 10.00 15.00 20.00 25.00
theta              -0.48 0.03 0.33  0.86  1.28  1.71  2.33
model_sem_raw       1.03 1.60 1.90  2.36  2.51  2.31  1.49
ctt_sem_raw         1.74 1.74 1.74  1.74  1.74  1.74  1.74
tcc_slope           2.36 5.70 7.67 11.04 12.33 10.90  4.84
sem_raw_over_slope  0.44 0.28 0.25  0.21  0.20  0.21  0.31
csem_theta          0.41 0.26 0.23  0.21  0.20  0.20  0.29

On the raw-score scale the model’s SEM is 2.36 at 10, 1.03 at 1 and 1.49 at 25, against CTT’s 1.74 everywhere. Divided by the expected-score curve’s slope (2.36 points per logit at a score of 1, 11.04 at 10), they become 0.44 and 0.21 on \(\theta\), close to the conditional SEMs of 0.41 and 0.21: narrow in points at the extremes, wide in \(\theta\).

Code 
# Bands on the theta scale: each respondent's estimate (EAP, mirt's default) plus or
# minus 1.96 conditional SEMs. How many positives have a band that reaches the cut?
th <- fscores(grm, verbose = FALSE)[, 1]
round(c(band_reaches_cut = mean(th[pos] - 1.96 * csem(th[pos]) <= theta_cut)), 3)
band_reaches_cut 
           0.562

With conditional-SEM bands on \(\theta\), 56.2% of the positives reach the cut, against 42.0% with CTT’s. Which should a report use? CTT’s band assumes the same error at every score; the model’s assumes its item curves and one \(\theta\) describe the data. At the cut the model’s SEM is about a third larger (2.36 points against 1.74). Whether the graded model’s assumptions hold well enough is a question the data can take up, for instance by checking its predictions of held-out responses (Model fit and out-of-sample prediction). Where they hold, the conditional SEM describes the error where the decision is made, and the one SEM summarizes the scale as a whole.

Change between waves

Code 
w2 <- long2wide(uk[uk$wave == 2, ])
both <- intersect(rownames(w1), rownames(w2))
x1 <- rowSums(w1[both, ]); x2 <- rowSums(w2[both, ])
r12 <- cor(x1, x2); a1 <- alpha(w1[both, ]); a2 <- alpha(w2[both, ])
# Reliability of the difference x2 - x1 (true-score variance of the difference over
# its observed variance)
rel_diff <- (a1 * var(x1) + a2 * var(x2) - 2 * cov(x1, x2)) / var(x2 - x1)
rci <- 1.96 * sqrt(2) * sem              # reliable change: |x2 - x1| at least this
dx <- x2 - x1
crossed <- (x1 >= 10) != (x2 >= 10)
round(c(n = length(both), r = r12, alpha_1 = a1, alpha_2 = a2, rel_diff = rel_diff,
        rci_threshold = rci, reliable_change = mean(abs(dx) >= rci),
        crossed_cut = mean(crossed), crossed_by_less = mean(crossed & abs(dx) < rci)), 3)
              n               r         alpha_1         alpha_2        rel_diff 
       1406.000           0.703           0.917           0.919           0.723 
  rci_threshold reliable_change     crossed_cut crossed_by_less 
          4.836           0.196           0.139           0.038

For the 1,406 seen twice, the waves correlate 0.70 and the change score has reliability 0.72. 19.6% changed by 4.8 points or more, the reliable change threshold. 13.9% crossed the cut, and 3.8% crossed it by less than 4.8 points: about a quarter of the crossings are within what error alone could produce. All this assumes the PHQ-9 measures the same thing at both waves, which Invariance and experience asks how to check.

Estimates and plausible values

Code 
# Estimates versus plausible values. EAPs are pulled toward the mean; plausible values
# are draws from each respondent's posterior, so across respondents they keep the
# spread of the population.
set.seed(61)
pv <- fscores(grm, plausible.draws = 20, verbose = FALSE)
round(c(sd_eap = sd(th), sd_pv = mean(sapply(pv, function(p) sd(p[, 1]))),
        above_cut_eap = mean(th >= theta_cut),
        above_cut_pv = mean(sapply(pv, function(p) mean(p[, 1] >= theta_cut))),
        observed_10_plus = mean(x >= 10)), 3)
          sd_eap            sd_pv    above_cut_eap     above_cut_pv 
           0.921            1.001            0.199            0.205 
observed_10_plus 
           0.221

The EAPs have an SD of 0.921 and put 19.9% above the cut; plausible values have an SD of 1.001 and put 20.5% above. The gap is small because the PHQ-9 is precise near the cut.

Problems

  1. Derivation. Show that for two occasions with equal variances, reliabilities \(\rho_1\) and \(\rho_2\) and correlation \(r_{12}\), the difference score has reliability \(\big((\rho_1 + \rho_2)/2 - r_{12}\big)/(1 - r_{12})\) (Lord, 1956). When is it zero? Can it be negative, and what would that say about the classical assumptions?
  2. Which SEM for change? Recompute the reliable change threshold with the wave-2 alpha, and with the test–retest correlation (0.70) in place of alpha. How many respondents change reliably under each? Which would you use, and why?
  3. Old norms. A clinic in 2026 reads patients’ PHQ-9 scores against the UK spring-2020 percentiles. What would you tell them? Use Standard 5.9 and what the German comparison showed about reference groups.
  4. A report for one respondent. Design a one-page PHQ-9 report for a respondent who scored 12: score, band, percentile rank with reference group and date, and the cut. What would you leave out, and why?
  5. Another reference group. Add kohlmann_2016_phq9, PHQ-9 responses from patients with coronary heart disease (Kohlmann et al., 2016). Where does a 10 fall there? Which reference group should a cardiologist use for a new patient, and for which question?
  6. Challenge (open): prevalence from a screener. The cut of 10 was chosen against diagnostic interviews. Should a population survey report the share scoring 10 or more as the prevalence of depression? Think about the positive predictive value at a low base rate. I don’t know of an agreed answer.

Stuck, or curious about something this lesson doesn’t cover? Ask Claude about this lesson opens a new conversation that already knows what the lesson is about; add your question at the end. Claude shows a caution notice on any prompt that arrives through a link; that’s expected.

Going further

  • AERA, APA, & NCME. (2014). Standards for educational and psychological testing. American Educational Research Association. Open access: full text (PDF); the open-access page. Chapter 5 covers scores, scales, norms and cut scores; chapter 2, standard errors near cuts and for differences.
  • Kolen, M. J., & Brennan, R. L. (2014). Score scales. In Test equating, scaling, and linking (3rd ed., pp. 371–485). Springer. https://doi.org/10.1007/978-1-4939-0317-7_9. Percentile ranks, normalized scores and the construction of score scales.
  • Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508–600). American Council on Education. https://openlibrary.org/works/OL3700878W. The classic chapter on norms, and the source of the Angoff method.
  • Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. SAGE. https://doi.org/10.4135/9781412985918. One chapter per standard-setting method, including Angoff (ch. 6) and bookmark (ch. 10).
  • Rogosa, D. R., & Willett, J. B. (1983). Demonstrating the reliability of the difference score in the measurement of change. Journal of Educational Measurement, 20(4), 335–343. https://doi.org/10.1111/j.1745-3984.1983.tb00211.x. When a difference score is reliable.
  • Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19. https://doi.org/10.1037/0022-006x.59.1.12. The reliable change index.
  • Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32(2), 179–197. https://doi.org/10.1111/j.1745-3984.1995.tb00462.x. Classification consistency without an item response model.
  • Lee, W.-C. (2010). Classification consistency and accuracy for complex assessments using item response theory. Journal of Educational Measurement, 47(1), 1–17. https://doi.org/10.1111/j.1745-3984.2009.00096.x. The IRT route of the Go deeper callout, for mixed item formats.
  • Levis, B., Benedetti, A., & Thombs, B. D. (2019). Accuracy of Patient Health Questionnaire-9 (PHQ-9) for screening to detect major depression: Individual participant data meta-analysis. BMJ, 365, l1476. https://doi.org/10.1136/bmj.l1476. How the PHQ-9’s cuts perform, with data from 58 studies.
  • Zenisky, A. L., & Hambleton, R. K. (2012). Developing test score reports that work: The process and best practices for effective communication. Educational Measurement: Issues and Practice, 31(2), 21–26. https://doi.org/10.1111/j.1745-3992.2012.00231.x. How to design a report that readers read correctly.
  • Wu, M. (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31(2–3), 114–128. https://doi.org/10.1016/j.stueduc.2005.05.005. What plausible values are for, with worked comparisons.
  • Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x. The PHQ-9’s first validation, with the cut of 10 and the severity bands.
  • Hambleton, R. K., & Zenisky, A. L. (2017). Score reporting and interpretation. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 3. Applications (pp. 127–142). Chapman and Hall/CRC. https://doi.org/10.1201/9781315117430. A handbook chapter on score reporting and interpretation.
  • Lewis, D., & Lord-Bessen, J. (2017). Standard setting. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 3. Applications (pp. 229–248). Chapman and Hall/CRC. https://doi.org/10.1201/9781315117430. A handbook chapter on standard setting.

Data sources

Data from the Item Response Warehouse (each table name links to its IRW page); references from IRW biblio.

  • c19prc_uk_mcbride_2021_phq9: Bentall, Richard and Shevlin, Mark and McBride, Orla and Murphy, Jamie and Hartman, Todd K and Levita, Liat and Gibson-Miller, Jilly and Mason, Liam and Bennett, Kate (2022). The COVID-19 Psychological Research Consortium Study, 2020-2021. [Data Collection]. Colchester, Essex: UK Data Service. 10.5255/UKDA-SN-855552. https://osf.io/v2zur/. Licence: CC BY 4.0.
  • coroiu_2018_phq9: Coroiu A, Kwakkenbos L, Moran C, Thombs B, Albani C, Bourkas S, et al. (2018). Structural validation of the Self-Compassion Scale with a German general population sample. PLOS ONE 13(2): e0190771. https://doi.org/10.1371/journal.pone.0190771. Licence: CC BY 4.0.

For instructors

  • Source: new material, not in EDUC 252: norms, error bands, change scores, cut scores and score reports, following chapters 2 and 5 of the Standards (AERA, APA & NCME, 2014, open access).
  • Timing: the core ideas and widgets take about 60 minutes; the PHQ-9 analysis works as a 30-minute lab. The graded model fits in a few seconds in local R.
  • Changed from the outline: the cut on \(\theta\) is where the test characteristic curve reaches 10. The Simulate section compares the model’s prediction with CTT’s normal approximation rather than with Livingston and Lewis’s beta-binomial model, which the Go deeper callout describes. The German cardiac sample appears only in problem 5.
  • Held back: the accuracy of a cut against a criterion (Validity evidence); norms on a new form, which need the forms linked first (Equating and linking); whether a unit means the same across the scale (Scale properties); precision at a cut in adaptive testing (Item banks and adaptive testing); estimating \(\theta\) for individuals (Estimating abilities). The fairness of cut scores across groups isn’t treated anywhere in the course.
  • Item text: the PHQ-9 is in the public domain: its instruction manual says no permission is required to reproduce, translate, display or distribute it (PHQ instruction manual, n.d., p. 8; the PHQ Screeners site says the same). The lesson still quotes only the first item.
  • Sensitivity: item 9 asks about thoughts of self-harm. The lesson doesn’t analyse it on its own; if students do, a reminder of where support is available is appropriate.
  • Solutions: held back in solutions/score-meaning.qmd (not published). Problem 6 is open.