Measurement invariance under treatment and life events

Module: Fairness and invariance
Tranche: extension (beyond a first course)
Before this: Differential item functioning
Builds toward: none
Picks up threads: A group difference is impact unless the measure works differently (from Fairness in testing: bias, impact, and measurement invariance); Items can function differently by treatment (from Differential item functioning)
IRW tables: gilbert_meta_20, gilbert_meta_37
From EDUC 252: problem sets PS5#4, PS7#1; code ps5/rctdif.R, ps7/ilhte.R
Status: draft
Authorship: Written largely by Claude (Anthropic), from Ben Domingue’s EDUC 252 materials and under Ben’s direction. Ben reviews each lesson before it is marked done.
Comments: open an issue on GitHub about this lesson.

What this is for

A fractions program for struggling fifth graders raises its students’ test scores by more than a standard deviation. Did it raise every item, or a handful? Differential item functioning ended on a vocabulary trial in which one word, expedition, carried over a quarter of the raw effect. This lesson takes that finding further. Experience, whether a program, the loss of a spouse or a second sitting of the same test, can change how a measure works, and a randomized trial is the one setting where we can separate that change from a change in the respondents. We read treatment as a grouping variable, use the pretest as a placebo, fit a model that gives every item its own treatment effect, and ask what the spread of those effects does to the average effect we report.

Goals

By the end of this lesson you should be able to:

  • say how experience can change a measure (alpha, beta and gamma change; response shift) as well as respondents’ standing on it;
  • use randomization to read treatment DIF as caused by the treatment, and a pretest as a placebo check on the whole analysis;
  • write down and fit the item-level heterogeneous treatment effects (IL-HTE) model with lme4, and interpret \(\sigma_\zeta\) and \(\rho\);
  • explain what item-level effects do to the precision and generalizability of an average effect, and how they can produce a treatment-by-pretest interaction that no respondent has;
  • say, for a given use of the scores, what would make the items that move most a threat or the signal.

Core ideas

When experience changes the measure

After a spouse dies, older adults in the Health and Retirement Study report more depressive symptoms on an eight-item, yes/no version of the CES-D (Radloff, 1977), and the rise fades within about a year (Domingue, Duncan, Harrati & Belsky, 2021). The items don’t share the rise equally: loneliness and sadness carry most of it (the preprint, Domingue et al., 2019, Supplemental Figures S7–S8). Is that more depression, or depression measured differently? A widow who now reports feeling lonely may be no more depressed, in the sense the other items capture, than a year earlier; the loneliness item is also answering a question about her circumstances. The IRW has no item-level data from this study, so the result stays a cited figure.

The possibility has names. Golembiewski, Billingsley and Yeager (1976) distinguished three kinds of change. Alpha change is a change in level on an instrument that stays the same; beta change, a change in the scale respondents use to answer; gamma change, a redefinition of the construct itself (Livingston et al., 2022, give a short account). In health research, Sprangers and Schwartz (1999) call the last two response shift: a change in what a self-evaluation means, from new internal standards, new values or a new conception of the construct. Each is a failure of invariance across occasions. Gamma change is a change in the factor structure; beta change, unequal loadings or intercepts (Jabrayilov, Emons, de Jong & Sijtsma, 2017; Oort, 2005; Widaman, Ferrer & Conger, 2010).

Recall from Fairness in testing: impact is a difference in the attribute itself; bias is a difference in scores that comes from something else. Invariance is always relative to the other items: a shift shared by every item looks like impact.

Swap “group” for “occasion” and the fairness lesson carries over: a change in level is impact across time, and an item whose meaning shifts is DIF across time. A program, a bereavement and a second sitting are all experiences between two measurements, and each can change the measure along with the person.

If you’ve done Measurement invariance: longitudinal invariance is the multigroup ladder (configural, metric, scalar) with occasions in place of groups. Generalizability theory treats the occasion as a facet, a source of error to average over; here it is a condition that may change what is measured. Factor analysis I used BFI-2 data in which students were randomly assigned a response format, the same design with the format as the treatment. And What does a score mean? computed a reliable change index, which assumes the scale measures the same thing at both waves (Jacobson & Truax, 1991).

Treatment as the grouping, and the pretest as a placebo

Recall from Differential item functioning: in a randomized trial, treatment assignment is a grouping too. Among children with the same total, the vocabulary item expedition was far easier for treated children, and DIF by treatment is variation in the treatment effect from item to item.

After a bereavement we can’t separate change in the person from change in the measure, since who is bereaved isn’t random. A randomized trial removes the problem. At the pretest, treated and control respondents come from the same population, so there is no impact and, since the program hasn’t happened yet, no reason for any item to work differently. Any treatment DIF found then is a false positive, and the pretest becomes a placebo test for the whole analysis, from the handling of waves to the matching score. At the posttest the arms differ only by assignment, so an item that works differently for treated respondents does so because of the treatment (Gilbert, Kim & Miratrix, 2023; Ahmed et al., 2025).

What the treatment causes is still relative. If the program raised every item by the same amount, the matched comparison would find nothing: that is impact, the average effect. Treatment DIF is each item’s departure from the average.

Try it: the pretest as a placebo

A simulated trial: respondents in each arm answer the same Rasch items twice. At the pretest nobody has been treated. At the posttest everyone has grown by 0.5 logits, treated respondents by a further 0.5 on average, and each item’s effect departs from that average with the spread you set. Each dot is an item’s Mantel–Haenszel delta by treatment (positive: easier for treated respondents with the same total), orange when \(p < .05\).

Step through the samples: the pretest flags hover around what chance allows, while the posttest, with the spread at 0.5, flags about a third of the items. Now set the spread to 0. Every item moves by the same amount and the posttest looks like the pretest again: a treatment that lifts all items equally shows up as impact, not DIF.

A treatment effect for every item

Rather than test items one at a time, we can model the variation. The IL-HTE model (Gilbert, Kim & Miratrix, 2023; Gilbert, Himmelsbach, Soland, Joshi & Domingue, 2025) is the Rasch model with persons and items both random (De Boeck & Wilson, 2004) and a random item slope on treatment, \(T_j\): \[\begin{aligned} \text{logit}\Pr(x_{ij} = 1) &= \theta_j - b_i + \zeta_i T_j,\\ \theta_j &= \beta_0 + \beta_1 T_j + \varepsilon_j, \qquad \varepsilon_j \sim N(0, \sigma_\theta^2),\\ \begin{pmatrix} b_i \\ \zeta_i \end{pmatrix} &\sim N\!\left(\begin{pmatrix} 0 \\ 0 \end{pmatrix}, \begin{pmatrix} \sigma_b^2 & \rho\sigma_b\sigma_\zeta \\ \rho\sigma_b\sigma_\zeta & \sigma_\zeta^2 \end{pmatrix}\right). \end{aligned}\] \(\beta_1\) is the average effect on \(\theta\); item \(i\)’s effect is \(\beta_1 + \zeta_i\). \(\sigma_\zeta\) is the spread of item effects around the average. At zero every item moves by \(\beta_1\), which is what a Rasch model fitted to both arms assumes: treatment changes \(\theta\) and nothing else. \(\rho\) says whether harder items (positive) or easier items (negative) move more. Dividing \(\beta_1\) and \(\sigma_\zeta\) by \(\sigma_\theta\) puts them in standard deviations of ability.

It is a generalized linear mixed model, so lme4 fits it (Bates, Mächler, Bolker & Walker, 2015; De Boeck et al., 2011):

glmer(resp ~ treat + (1 | id) + (treat | item), data, family = binomial)

lme4’s item intercept is an easiness, \(-b_i\), so the correlation it prints is \(-\rho\). Is \(\sigma_\zeta\) zero? Compare this model with one that has (1 | item) by a likelihood ratio. A variance can’t be negative, so the null value sits on the boundary of the parameter space, and halving the \(\chi^2\) \(p\)-value is the usual correction (Self & Liang, 1987).

Try it: item effects around an average

Twenty items with difficulties spread around the average respondent. Set the average effect \(\beta_1\), the spread \(\sigma_\zeta\) and the correlation \(\rho\) between difficulty and effect. The five items that respond most are orange.

With \(\sigma_\zeta\) at 0 every dot sits on the dashed line, and a test built from any five items would show the same effect in logits. At the default, the five most responsive items average more than twice the overall effect and the five least about zero: two short forms of one test, one program, two conclusions. Move \(\rho\) to −0.8 and the orange dots move left, among the easy items.

What item-level effects do to the estimate

Does the spread change the average? Not by itself: the sum score, a constant-effect model and the IL-HTE model estimate much the same average, as the fractions trial below shows. It changes three other things.

Precision. The items are a sample from those that could have been written, and if they respond unequally, a different sample would give a different average. Treating the items as fixed leaves that variation out, which understates standard errors and inflates false positives (Gilbert, Kim & Miratrix, 2023). Across 15 trials, Ahmed et al. (2025) found one item driving nearly 40% of an effect, and item-level patterns that could make 31% of the significant effects null once counted as uncertainty.

Generalizability. \(\sigma_\zeta\) says how far the effect on a fresh set of items from the same domain could fall from the one observed. A claim about the construct, rather than about these items, needs it (Gilbert, Himmelsbach, et al., 2025).

Recall from Validity as an argument: an interpretation/use argument has a generalization link, from the observed score to the score over all the tasks it stands for. \(\sigma_\zeta\) is evidence on that link for an effect rather than a score.

Interactions. Trialists ask who benefits most by interacting treatment with a pretest. Suppose every respondent gains the same amount on \(\theta\), but easier items respond more. In the treated arm the items’ effective difficulties, \(b_i - \zeta_i\), are more spread out than in the control arm, the sum score rises less steeply with \(\theta\), and a sum-score regression finds a negative treatment-by-pretest interaction that no respondent has. From sum scores alone the two stories can’t be told apart; item-level data can tell them apart (Gilbert, Miratrix, Joshi & Domingue, 2025).

Try it: an interaction nobody has

Treatment adds 0.5 logits to every respondent’s \(\theta\), whatever their pretest; the 20 items’ effects have the spread and correlation you set. The curves are the expected sum score at each pretest level in each arm, computed from the model, so there is no sampling noise.

With every item moving equally the interaction is already slightly negative, from the sum score’s bounded range. At the default, where easier items move more, it is five times that, with no heterogeneity among respondents at all. Set \(\rho\) to +0.8 and it changes sign.

Assumptions: the IL-HTE model above; no person-level heterogeneity (every respondent’s \(\theta\) rises by \(\beta_1\)); the outcome analysed is the sum score \(S = \sum_i x_i\).

Two tests in one. In the control arm item \(i\) has difficulty \(b_i\). In the treated arm it behaves as if its difficulty were \(b_i - \zeta_i\), on top of the shift of \(\beta_1\) in \(\theta\). The spread of effective difficulties in the treated arm is \[\text{Var}(b_i - \zeta_i) = \sigma_b^2 + \sigma_\zeta^2 - 2\rho\sigma_b\sigma_\zeta,\] which exceeds \(\sigma_b^2\) whenever \(\rho \le 0\), and can fall below it when \(\rho\) is positive and large enough.

What spread does to the sum score. The expected sum score is the test characteristic curve, \(E(S \mid \theta) = \sum_i P_i(\theta)\) with \(P_i(\theta) = \text{logit}^{-1}(\theta - b_i)\). Its slope at \(\theta\) is \(\sum_i P_i(1 - P_i)\), the test information under the Rasch model. An item near \(\theta\) contributes close to the maximum of 1/4; spreading the items out moves some of them where they contribute little. So in the middle of the ability range the treated arm’s curve is flatter when \(\rho < 0\).

The interaction. A regression of \(S\) on treatment, pretest and their product estimates the slope of \(S\) on the pretest in each arm. The pretest predicts \(\theta\) equally well in both arms, but the treated arm’s flatter curve turns a unit of \(\theta\) into fewer points, so its slope is smaller and the interaction is negative, although every respondent gained the same. Gilbert, Miratrix, Joshi and Domingue (2025) show analytically and by simulation that a sum score cannot distinguish this from a true interaction, and that a model with item-specific effects can. The bounded range of \(S\) adds a smaller effect of its own, which is why the widget’s interaction is not quite zero when \(\sigma_\zeta = 0\).

Bias, or what the program taught?

Is an item that moves most a biased item? It may be the item closest to what an intervention taught. For evaluating the intervention, that can be the signal: the program changed what it set out to change, and the near items show it. For a claim of general gain, it is a threat: a test made mostly of near items overstates transfer, in the sense of Barnett and Ceci’s (2002) taxonomy of near and far transfer. Whether item-level heterogeneity is bias depends on the interpretation and the use, as Validity as an argument would lead you to expect. Group-specific item parameters can also be judged by prediction: do they predict new responses better than shared ones? The InterModel Vigorish measures that (Domingue et al., 2024). Explaining which items move, with features such as alignment to the curriculum, is a job for What is an item?.

Simulate

The code simulates a trial with 800 respondents and 20 items. Treatment raises every respondent’s \(\theta\) by 0.5, whatever their pretest; the items’ departures have \(\sigma_\zeta = 0.8\) and \(\rho = -0.8\) (easier items move more). It fits a sum-score regression, a constant-effect model and the IL-HTE model, each with treatment, pretest and their product. Loading lme4 takes 10 to 20 seconds the first time the page runs code; the fits take a few seconds more.

With the seed as written, the IL-HTE model recovers \(\sigma_\zeta\) = 0.78 and \(\rho\) = −0.78, and its item effects correlate 0.99 with the true ones. The interaction, 0 in truth, is −0.08 SD in the sum-score regression and −0.10 logits in the constant-effect model; the IL-HTE model puts it at 0.04. Its standard error is about 0.06 in both models, so a single run is noisy: try other seeds. Set rho <- 0.8 and the constant-effect interaction turns positive (0.03); set sigma_zeta <- 0 and the three models agree.

Download this code to run it with more respondents, or many replications, in your own R session.

With real data

Download the code; it needs lme4, and the model fits take about a minute.

A fractions trial, before and after

Our main table is gilbert_meta_20, from a randomized trial of a small-group fractions intervention for fifth graders performing well below grade level (Jayanthi et al., 2021; data at doi:10.7910/DVN/Z4WMFE), part of the trial collection of Gilbert, Himmelsbach, et al. (2025). Its job is to be a trial with a pretest, so the placebo check is real. Students who scored between the 15th and 37th percentiles on a fractions screener were randomized within teacher, and the table holds the final analytic sample of 186 (data description). Items are scored 1 for correct.

Code 
# The IRW table, from the CSV link on its landing page (pinned to one version).
url20 <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v60_0.gilbert_meta_20/rows?format=csv"
f20 <- read.csv(url20)
f20 <- f20[!is.na(f20$resp), ]
# Waves: 0 = pretest, 1 = posttest. How many items and students in each?
t(sapply(split(f20, f20$wave), function(x) c(items = length(unique(x$item)),
                                            students = length(unique(x$id)))))
  items students
0    37      186
1    93      186
Code 
pre_items <- unique(f20$item[f20$wave == 0])
post_items <- unique(f20$item[f20$wave == 1])
c(pretest_items_also_at_posttest = sum(pre_items %in% post_items))
pretest_items_also_at_posttest 
                            37
Code 
# Items come in four blocks (by the prefix of their names):
table(wave = f20$wave[!duplicated(f20[, c("wave", "item")])],
      block = sub("_+q.*$", "", f20$item[!duplicated(f20[, c("wave", "item")])]))
    block
wave procedures trans_math tuf4 tuf5
   0         11          0   26    0
   1         24         25   26   18
Code 
kids20 <- f20[!duplicated(f20$id), c("id", "treat")]
table(treat = kids20$treat)
treat
 0  1 
99 87

On waves: wave 0 is the pretest and wave 1 the posttest. The posttest repeats all 37 pretest items (26 from the fourth-grade Test for Understanding of Fractions, tuf4, and 11 fractions procedures) and adds 56: the fifth-grade form (tuf5), more procedures, and a block the dataset labels TransMath (trans_math). Eighteen posttest responses are missing, all from one student, who drops out of the complete-case analyses.

We match on the total score and test uniform DIF, the kind a shift in one item’s treatment effect produces, with the logistic regression of the DIF lesson (Swaminathan & Rogers, 1990). With 37 or 93 tests, we also control the false discovery rate (Benjamini & Hochberg, 1995), and give the \(\Delta R^2\) categories of Jodoin and Gierl (2001).

Code 
# One wide matrix per wave: a row per student, a column per item (1 = correct).
wide <- function(long, items) {
  x <- long[long$item %in% items, ]
  A <- tapply(x$resp, list(x$id, x$item), function(v) v[1])
  A <- A[complete.cases(A), items]
  list(X = A, treat = kids20$treat[match(rownames(A), kids20$id)])
}
# Logistic-regression DIF for one item, matching on the total score, as in the DIF
# lesson but for uniform DIF only, the kind a shift in one item's treatment effect
# produces: the 1-df likelihood-ratio test of M0 (total) against M1 (total and
# treatment), and the change in Nagelkerke's R^2 between them.
nagelkerke <- function(m, ll0, n) {
  cs <- 1 - exp(2 * (ll0 - as.numeric(logLik(m))) / n)
  cs / (1 - exp(2 * ll0 / n))
}
lr_item <- function(y, g, total) {
  d <- data.frame(y = y, g = g, total = total)
  ll0 <- as.numeric(logLik(glm(y ~ 1, binomial, d)))
  m0 <- glm(y ~ total, binomial, d)
  m1 <- glm(y ~ total + g, binomial, d)
  c(beta_treat = unname(coef(m1)["g"]),
    p = anova(m0, m1, test = "LRT")[2, "Pr(>Chi)"],
    dR2 = nagelkerke(m1, ll0, nrow(d)) - nagelkerke(m0, ll0, nrow(d)))
}
# Every item; p-values also after the Benjamini-Hochberg false-discovery-rate
# correction; Jodoin-Gierl categories (A below .035, B to .07, C above, and A
# whenever the test is not significant).
treatment_dif <- function(W) {
  tot <- rowSums(W$X)
  out <- as.data.frame(t(sapply(colnames(W$X), function(i) lr_item(W$X[, i], W$treat, tot))))
  out$p_BH <- p.adjust(out$p, "BH")
  out$jg <- ifelse(out$p >= 0.05 | out$dR2 < 0.035, "A", ifelse(out$dR2 < 0.07, "B", "C"))
  out
}
count_flags <- function(r) c(items = nrow(r), p_below_05 = sum(r$p < 0.05),
                             expected_by_chance = round(0.05 * nrow(r), 1),
                             after_BH = sum(r$p_BH < 0.05), B = sum(r$jg == "B"), C = sum(r$jg == "C"))

The pretest is also our sanity table: before treatment, its flags estimate the false-positive rate. It has 3 flags at .05 among 37 items, about what chance allows, and none survives the correction (the table below).

Code 
W0 <- wide(f20[f20$wave == 0, ], pre_items)    # pretest: before any instruction
W1 <- wide(f20[f20$wave == 1, ], post_items)   # posttest: all 93 items
W1s <- wide(f20[f20$wave == 1, ], pre_items)   # posttest, the pretest's 37 items only
dif0 <- treatment_dif(W0); dif1 <- treatment_dif(W1); dif1s <- treatment_dif(W1s)
rbind(pretest = count_flags(dif0), posttest = count_flags(dif1),
      "posttest, same 37 items" = count_flags(dif1s))
                        items p_below_05 expected_by_chance after_BH B C
pretest                    37          3                1.9        0 3 0
posttest                   93         12                4.7        5 8 2
posttest, same 37 items    37          9                1.9        5 3 1

With 186 students the \(\Delta R^2\) categories are generous: all three pretest flags reach B, so here the correction is the better guide. After the program the same 37 items give 9 flags, 5 of which survive; across all 93, 12 and 5, with 8 items in B and 2 in C.

Code 
# The posttest items in B or C, largest first. beta_treat > 0: easier for treated
# students with the same total.
d1 <- dif1[dif1$jg != "A", ]
d1$p_control <- colMeans(W1$X[W1$treat == 0, rownames(d1)])
d1$p_treated <- colMeans(W1$X[W1$treat == 1, rownames(d1)])
round(d1[order(-d1$dR2), c("beta_treat", "p", "dR2", "p_control", "p_treated")], 3)
beta_treat p dR2 p_control p_treated
trans_math__q10 1.287 0.000 0.083 0.384 0.767
tuf4_q21 -1.466 0.001 0.072 0.333 0.337
procedures__q1h -1.803 0.004 0.066 0.111 0.151
tuf4_q04 1.133 0.005 0.052 0.556 0.872
trans_math__q2 0.996 0.006 0.050 0.515 0.802
procedures__q1b 1.266 0.002 0.042 0.172 0.628
procedures__q2d 1.617 0.001 0.039 0.172 0.674
procedures__q2b 1.546 0.001 0.038 0.212 0.721
tuf4_q07 -0.860 0.020 0.038 0.374 0.326
tuf5_q02 -0.937 0.014 0.036 0.515 0.570
Code 
table(block = sub("_+q.*$", "", rownames(d1)), sign = ifelse(d1$beta_treat > 0, "favours treated", "favours control"))
            sign
block        favours control favours treated
  procedures               1               3
  trans_math               0               2
  tuf4                     2               1
  tuf5                     1               0

Most items that favour treated students come from the procedures and TransMath blocks; several from the understanding tests (tuf4, tuf5) go the other way, moved less than the rest of the test.

The IL-HTE model summarizes all of this in a few numbers:

Code 
suppressPackageStartupMessages(library(lme4))
# The IL-HTE model (Gilbert, Kim & Miratrix, 2023), in lme4's parameterization:
#   logit P(x_ij = 1) = beta0 + beta1 T_j + theta_j + e_i + zeta_i T_j,
# with theta_j ~ N(0, s_theta^2) and (e_i, zeta_i) bivariate normal. lme4's item
# intercept e_i is an easiness, the negative of the lesson's difficulty b_i, so the
# correlation it reports has the opposite sign to rho in the lesson; we flip it.
# Following the IRW vignette, effects are standardized by s_theta from the model with
# a constant effect, and sigma_zeta is tested by a likelihood-ratio test against
# that model, halving the p-value because a variance of zero sits on the boundary
# (Self & Liang, 1987).
ilhte <- function(long) {
  m0 <- glmer(resp ~ treat + (1 | id) + (1 | item), long, family = binomial)
  m1 <- glmer(resp ~ treat + (1 | id) + (treat | item), long, family = binomial)
  s_theta <- sqrt(VarCorr(m0)$id[1])
  vc <- VarCorr(m1)$item
  lr <- as.numeric(2 * (logLik(m1) - logLik(m0)))
  list(m0 = m0, m1 = m1, s_theta = s_theta,
       summary = c(students = length(unique(long$id)), items = length(unique(long$item)),
                   effect_constant_SD = unname(fixef(m0)["treat"]) / s_theta,
                   effect_average_SD = unname(fixef(m1)["treat"]) / s_theta,
                   sigma_zeta_SD = sqrt(vc[2, 2]) / s_theta,
                   rho_difficulty = -attr(vc, "correlation")[1, 2],
                   LR = lr, p = 0.5 * pchisq(lr, 2, lower.tail = FALSE)))
}
Code 
fit0 <- ilhte(f20[f20$wave == 0, ])
fit1 <- ilhte(f20[f20$wave == 1, ])
fit1s <- ilhte(f20[f20$wave == 1 & f20$item %in% pre_items, ])
tab <- rbind(pretest = fit0$summary, posttest = fit1$summary,
             "posttest, same 37 items" = fit1s$summary)
data.frame(round(tab[, 1:7], 2), p = sprintf("%.2g", tab[, "p"]))
students items effect_constant_SD effect_average_SD sigma_zeta_SD rho_difficulty LR p
pretest 186 37 0.18 0.24 0.59 0.43 1.65 0.22
posttest 186 93 1.11 1.13 0.70 0.05 140.41 1.6e-31
posttest, same 37 items 186 37 1.19 1.23 0.83 0.21 97.58 3.2e-22
Code 
# sigma_zeta in logits, and the SD of theta it is divided by:
round(sapply(list(pretest = fit0, posttest = fit1), function(f)
  c(sigma_zeta_logits = sqrt(VarCorr(f$m1)$item[2, 2]), s_theta = f$s_theta)), 2)
                  pretest posttest
sigma_zeta_logits    0.18     0.59
s_theta              0.31     0.84

At the pretest \(\sigma_\zeta\) is not distinguishable from zero (\(p\) = 0.22). Its 0.59 SD looks large only because \(\sigma_\theta\) is small there: the screener selected students from a narrow band, so a logit is a large share of an SD (in logits, \(\sigma_\zeta\) is 0.18 at the pretest and 0.59 at the posttest). At the posttest the average effect is 1.13 SD, \(\sigma_\zeta\) is 0.70 SD and the likelihood ratio is 140. On the 37 repeated items alone, \(\sigma_\zeta\) is 0.83 SD.

Code 
# The same posttest effect three ways, each in within-arm SDs: the sum score, and
# theta from the constant-effect and IL-HTE models.
tot1 <- rowSums(W1$X)
gap <- mean(tot1[W1$treat == 1]) - mean(tot1[W1$treat == 0])
sd_within <- sqrt(mean(tapply(tot1, W1$treat, var)))   # pooled within-arm SD, like s_theta
sum_effect <- gap / sd_within
round(c(sum_score = sum_effect, theta_constant = unname(fit1$summary["effect_constant_SD"]),
        theta_ilhte = unname(fit1$summary["effect_average_SD"])), 2)
     sum_score theta_constant    theta_ilhte 
          1.08           1.11           1.13
Code 
# Item-specific effects (beta1 + zeta_i, in SDs of theta), by block.
eff1 <- (fixef(fit1$m1)["treat"] + ranef(fit1$m1)$item[, "treat"]) / fit1$s_theta
names(eff1) <- rownames(ranef(fit1$m1)$item)
round(t(sapply(split(eff1, sub("_+q.*$", "", names(eff1))), function(e)
  c(items = length(e), mean = mean(e), min = min(e), max = max(e)))), 2)
           items mean   min  max
procedures    24 1.43  0.39 2.49
trans_math    25 1.14  0.24 2.14
tuf4          26 1.01 -0.64 2.19
tuf5          18 0.85  0.01 1.91

The sum score, \(\theta\) with a constant effect and \(\beta_1\) from the IL-HTE model agree on the average: 1.08, 1.11 and 1.13 SD. The spread sits underneath. Procedures items moved most, 1.43 SD on average against 0.85 for the fifth-grade understanding form, as the paper’s own effects by measure would suggest: largest for fractions procedures (\(g\) = 1.07), 0.66 to 0.78 for proficiency and understanding (Jayanthi et al., 2021, abstract). Within every block, item effects span more than a standard deviation.

Health knowledge in India

For contrast, gilbert_meta_37: health knowledge ten months after an entertainment-education programme in India. Respondents in the programme watched short films on hygiene, midwives and child health, condoms and AIDS, and night blindness; the control arm watched films on financial planning (Carpena, 2024, Supplementary Material A; data at doi:10.7910/DVN/EJ03JR). Its job: item-level effects whose content we can read. Items are knowledge questions scored 1 for a correct answer, and several are options of one select-all-that-apply question (ways to contract AIDS, times to wash hands). The table’s treat pools the paper’s two film arms, with and without cash incentives for the test, against the control arm. There is one wave, and nothing is missing.

Code 
url37 <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v60_0.gilbert_meta_37/rows?format=csv"
f37 <- read.csv(url37)
p37 <- f37[!duplicated(f37$id), ]
c(respondents = nrow(p37), items = length(unique(f37$item)), missing = sum(is.na(f37$resp)))
respondents       items     missing 
        839          21           0
Code 
table(treat = p37$treat)
treat
  0   1 
414 425
Code 
fit37 <- ilhte(f37)
data.frame(t(round(fit37$summary[1:7], 2)), p = sprintf("%.2g", fit37$summary[["p"]]))
students items effect_constant_SD effect_average_SD sigma_zeta_SD rho_difficulty LR p
839 21 0.49 0.48 0.28 0.33 15.7 0.00019
Code 
# Item-specific effects in logits (beta1 + zeta_i), with the proportion correct in
# each arm, largest effect first.
re <- ranef(fit37$m1)$item
items37 <- data.frame(effect_logits = fixef(fit37$m1)["treat"] + re[, "treat"],
                      p_control = tapply(f37$resp[f37$treat == 0], f37$item[f37$treat == 0], mean)[rownames(re)],
                      p_treated = tapply(f37$resp[f37$treat == 1], f37$item[f37$treat == 1], mean)[rownames(re)],
                      row.names = rownames(re))
round(items37[order(-items37$effect_logits), ], 2)
effect_logits p_control p_treated
nightblindness_nutrition 0.81 0.12 0.27
nightblindness_heard 0.75 0.61 0.78
nightblindness_curable 0.68 0.46 0.63
aids_pregnant 0.61 0.02 0.04
pregnancy_breastmilk 0.53 0.62 0.73
aids_needle 0.50 0.23 0.33
aids_pregnant_baby 0.45 0.25 0.34
condom_hiv 0.43 0.83 0.89
pregnancy_dai_certificate 0.41 0.69 0.77
aids_sex 0.39 0.71 0.77
diar_know 0.34 0.31 0.38
aids_heard 0.31 0.86 0.88
wash_defecation 0.31 0.91 0.93
aids_touch 0.30 0.84 0.87
condom_std 0.29 0.85 0.87
aids_blood 0.29 0.21 0.25
wash_eating 0.29 0.86 0.88
aids_curable 0.24 0.46 0.49
condom_disease 0.19 0.20 0.20
wash_cooking 0.17 0.36 0.37
condom_pregnancy 0.12 0.93 0.91

The average effect is 0.48 SD and \(\sigma_\zeta\) is 0.28 SD (\(p\) = 0.00019), a much smaller spread than in the fractions trial. The three items that moved most are the three about night blindness (0.68 to 0.81 logits), the subject of one of the programme’s five modules: a single film about a boy who can’t see at dusk (Supplementary Material A). The paper’s own results by topic agree: night blindness had the largest effect of the four topics, and the lowest knowledge in the control arm (Carpena, 2024, longer-term knowledge by topic). Knowing to wash hands before cooking and knowing that condoms prevent pregnancy moved least (0.17 and 0.12 logits). \(\rho\) = 0.33: harder items moved somewhat more.

Code 
# Who gains most? The treatment-by-baseline interaction, with a constant effect for
# every item and with IL-HTE (std_baseline: the IRW's standardized baseline score).
mB <- glmer(resp ~ treat * std_baseline + (1 | item) + (1 | id), f37, family = binomial)
mC <- glmer(resp ~ treat * std_baseline + (treat | item) + (1 | id), f37, family = binomial)
round(rbind(constant_items = summary(mB)$coef["treat:std_baseline", 1:2],
            ilhte = summary(mC)$coef["treat:std_baseline", 1:2]), 3)
               Estimate Std. Error
constant_items    0.024      0.065
ilhte             0.020      0.066

Does IL-HTE change the answer to “who gained most?” here? Barely: the treatment-by-baseline interaction is 0.024 logits with a constant effect for every item and 0.020 with IL-HTE, both well inside their standard errors. With a spread of 0.28 SD, against 0.70 in the fractions trial, and a moderate \(\rho\), there is little for the constant-effect model to misread.

What should a trialist take from this? Two positions rest on different assumptions. If the outcome’s items are the target (the state test a program will be judged on), the average over these items is the estimand, and the sum score answers it. If the items stand for a domain (fractions, health knowledge), the effect on the domain is the estimand, and the spread of item effects is part of its uncertainty. Fitting the IL-HTE model beside the sum-score analysis shows what the data can support under each. In the fractions trial the three analyses agree on the average (1.08, 1.11 and 1.13 SD), the pretest placebo passes, and items on the same test moved by amounts more than a standard deviation apart. Whether item-specific effects are worth their parameters can also be put to held-out responses, with the IMV of Model fit and out-of-sample prediction. An average effect is a claim about items as well as people.

Problems

  1. Raw effects and difficulty. Under the IL-HTE model with every \(\zeta_i = 0\), and \(\theta \sim N(0, 1)\) in control and \(N(\beta_1, 1)\) in the treated arm, write the effect on item \(i\)’s expected proportion correct as a function of \(b_i\). Show that it is largest for items near the average ability and small for very easy or very hard items. Why, then, can raw differences in proportion correct and item effects on the logit scale rank the items differently, and which would you use to say which items a program moved?
  2. How much do the flagged items carry? (PS5#4). In gilbert_meta_20 at the posttest, drop the 10 items in B or C and re-estimate the effect on the sum score and on \(\theta\). How much of the effect did they carry? Now drop instead the 10 items with the smallest estimated IL-HTE effects. Which short form would a program developer prefer, and which would you report?
  3. The interaction, with and without item slopes (PS7#1). In gilbert_meta_37, estimate the treatment-by-baseline interaction on the sum score, with a constant-effect model and with IL-HTE. From the IL-HTE fit, compute the SD of the items’ effective difficulties in each arm. Using the Go deeper, explain why the three estimates are so close here.
  4. Bereavement. After spousal loss, the CES-D items on loneliness and sadness carry most of the rise in symptoms (Domingue et al., 2019). Set out what you would have to assume to report “depression rose”, and what you would have to assume to report “the CES-D measured something different”. Given item-level data from waves before and after the loss, which model would you fit, and what result would lead you to each report?
  5. Near and far. You are designing the outcome measure for a vocabulary intervention that teaches 40 words. Design a test that can separate near transfer (taught words) from far transfer (untaught words of similar frequency), and say how you would model item-level effects so that the two can be compared. What would you report if the near items moved and the far items didn’t?
  6. Challenge: evidence for or against? Suppose the items that move most in a trial are exactly those most aligned with the intervention’s content. Is item-level heterogeneity then evidence against the measure’s validity for evaluating the intervention, evidence for it, or neither? Does your answer change if the same test is also used to report students’ general proficiency? This one is open.

Stuck, or curious about something this lesson doesn’t cover? Ask Claude about this lesson opens a new conversation that already knows what the lesson is about; add your question at the end. Claude shows a caution notice on any prompt that arrives through a link; that’s expected.

Across the IRW

How common are item-level treatment effects across the trials in the warehouse? Following the IRW’s IL-HTE vignette, compute.R takes every table with a treatment indicator and at least 100 respondents, keeps the dichotomous items (at least 3), samples at most 5,000 respondents, and fits this lesson’s two models. It departs from the vignette on waves: the outcome is the first wave after randomization, read from the processing notes where they have been checked and otherwise by a stated rule, and any pretest wave is fitted too, as a placebo.

Code 
dd_path <- file.path("..", "deepdives", "invariance-experience", "results.rds")
dd <- if (file.exists(dd_path)) readRDS(dd_path) else NULL
if (!is.null(dd)) {
  s <- dd$summary[order(-dd$summary$sigma_zeta_sd), ]
  print(data.frame(table = s$table, wave = s$wave, respondents = s$n_persons, items = s$n_items,
                   effect_SD = round(s$ate_sd, 2), sigma_zeta_SD = round(s$sigma_zeta_sd, 2),
                   rho = round(s$rho, 2), p = signif(s$p, 2)), row.names = FALSE)
  if (!is.null(dd$placebo)) {
    cat("\nPlacebo (pretest waves):\n")
    print(data.frame(table = dd$placebo$table, wave = dd$placebo$wave,
                     sigma_zeta_SD = round(dd$placebo$sigma_zeta_sd, 2), p = signif(dd$placebo$p, 2)),
          row.names = FALSE)
  }
  if (nrow(dd$failed)) { cat("\nFailed:\n"); print(dd$failed, row.names = FALSE) }
}
           table wave respondents items effect_SD sigma_zeta_SD   rho       p
 gilbert_meta_20    1         186    93      1.13          0.70  0.05 1.6e-31
 gilbert_meta_37   NA         839    21      0.48          0.28  0.33 1.9e-04
 gilbert_meta_89    1        1789    80     -0.02          0.02 -1.00 4.4e-01

Placebo (pretest waves):
           table wave sigma_zeta_SD    p
 gilbert_meta_20    0          0.59 0.22
 gilbert_meta_89    0          0.01 0.47

Failed:
                          table                          error
               wekker_2018_mfsq fewer than 3 dichotomous items
 buczel_2022_inoculation_belief fewer than 3 dichotomous items
WarningPartial run: 3 of 157 tables

This is a pilot run of compute.R, on two tables whose answers are known (gilbert_meta_20 and gilbert_meta_37) and 3 others drawn at random, to check the pipeline. The full run replaces it.

Run on 25 September 2026: 3 tables analysed, 2 failed. Item-level variation is significant at .05 in 2 of the 3.

The pilot checks the pipeline. gilbert_meta_20 and gilbert_meta_37 reproduce this lesson’s estimates, and gilbert_meta_20’s pretest again shows no item-level variation. In gilbert_meta_89 the arms end up level, and so do the items: the average effect and \(\sigma_\zeta\) are both near zero, at the pretest and after. The two tables that failed have no dichotomous items. The full run will say how often items respond unequally across the IRW’s trials, and whether the pretest placebo passes where there is one. The vignette itself is the place to read more.

Going further

  • Gilbert, J. B., Kim, J. S., & Miratrix, L. W. (2023). Modeling item-level heterogeneous treatment effects with the explanatory item response model: Leveraging large-scale online assessments to pinpoint the impact of educational interventions. Journal of Educational and Behavioral Statistics, 48(6), 889–913. https://doi.org/10.3102/10769986231171710. The IL-HTE model, and what ignoring it does to standard errors.
  • Gilbert, J. B., Himmelsbach, Z., Soland, J., Joshi, M., & Domingue, B. W. (2025). Estimating heterogeneous treatment effects with item-level outcome data: Insights from item response theory. Journal of Policy Analysis and Management, 44(4), 1417–1449. https://doi.org/10.1002/pam.70025. The model across 75 datasets from 48 trials, the collection this lesson’s tables come from.
  • Gilbert, J. B., Miratrix, L. W., Joshi, M., & Domingue, B. W. (2025). Disentangling person-dependent and item-dependent causal effects: Applications of item response theory to the estimation of treatment effect heterogeneity. Journal of Educational and Behavioral Statistics, 50(1), 72–101. https://doi.org/10.3102/10769986241240085. The interaction of the Go deeper, derived and simulated.
  • Ahmed, I., Bertling, M., Zhang, L., Ho, A. D., Loyalka, P., Xue, H., Rozelle, S., & Domingue, B. W. (2025). Heterogeneity of item-treatment interactions masks complexity and generalizability in randomized controlled trials. Journal of Research on Educational Effectiveness, 18(4), 854–877. https://doi.org/10.1080/19345747.2024.2361337. DIF by treatment across 15 trials.
  • Sprangers, M. A. G., & Schwartz, C. E. (1999). Integrating response shift into health-related quality of life research: A theoretical model. Social Science & Medicine, 48(11), 1507–1515. https://doi.org/10.1016/S0277-9536(99)00045-3. Response shift, defined, with a model of how it arises.
  • Oort, F. J. (2005). Using structural equation modeling to detect response shifts and true change. Quality of Life Research, 14(3), 587–598. https://doi.org/10.1007/s11136-004-0830-y. Response shift as non-invariance across occasions.
  • Widaman, K. F., Ferrer, E., & Conger, R. D. (2010). Factorial invariance within longitudinal structural equation models: Measuring the same construct across time. Child Development Perspectives, 4(1), 10–18. https://doi.org/10.1111/j.1750-8606.2009.00110.x. Longitudinal invariance in a few pages.
  • Fischer, G. H. (2017). Measuring change using Rasch models. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 3. Applications (pp. 387–404). Chapman and Hall/CRC. https://doi.org/10.1201/9781315117430. A handbook chapter on measuring change with Rasch models.
  • De Boeck, P., & Wilson, M. (Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. Springer. https://doi.org/10.1007/978-1-4757-3990-9. Item response models as mixed models, the frame the IL-HTE model sits in.
  • Domingue, B. W., Duncan, L., Harrati, A., & Belsky, D. W. (2021). Short-term mental health sequelae of bereavement predict long-term physical health decline in older adults: U.S. Health and Retirement Study analysis. The Journals of Gerontology: Series B, 76(6), 1231–1240. https://doi.org/10.1093/geronb/gbaa044. The spousal-loss example; the item-level figures are in the supplement to the preprint (https://doi.org/10.1101/19009878).
  • From the IRW: the IL-HTE vignette, the corpus question of Across the IRW.

Data sources

Data from the Item Response Warehouse (each table name links to its IRW page); references from IRW biblio.

  • gilbert_meta_20: Jayanthi, M., Gersten, R., Schumacher, R. F., Dimino, J., Smolkowski, K., & Spallone, S. (2021). Improving struggling fifth-grade students’ understanding of fractions: A randomized controlled trial of an intervention that stresses both concepts and procedures. Exceptional Children, 88(1), 81-100. Gersten, Russell; Jayanthi, Madhavi; Dimino, Joseph; Karp, Karen, 2019, “Impact of a Tier 2 Fractions Intervention on 5th Grade At-Risk Students’ Fractions Achievement”, https://doi.org/10.7910/DVN/Z4WMFE, Harvard Dataverse, V1, UNF:6:8pMQK/5+cfpvcr4ACTvAmQ== [fileUNF]. Licence: CC0 1.0.
  • gilbert_meta_37: Carpena, F. (2024). Entertainment-Education for Better Health: Insights from a Field Experiment in India. The Journal of Development Studies, 60(5), 745-762. Carpena, Fenella, 2024, “Replication Data for: Entertainment-Education for Better Health: Insights from a Field Experiment in India”, https://doi.org/10.7910/DVN/EJ03JR, Harvard Dataverse, V1, UNF:6:pX0ZhtBxkRpvQC6AOfVg4g== [fileUNF]. Licence: CC0 1.0.

For instructors

  • Source: EDUC 252 problem sets 5 (#4: treatment DIF in gilbert_meta_20, and the spousal-loss question) and 7 (#1: the treatment-by-baseline interaction with and without random item slopes), with the code in ben-domingue/252: ps5/rctdif.R and ps7/ilhte.R (which used gilbert_meta_2). There are no slides; the widgets, the simulation and the two-table analysis are new.
  • Timing: about 50 minutes for the core ideas and widgets; the real-data section works as a 30-minute lab (the model fits take about a minute locally). Problems 4 and 6 work as small-group discussions.
  • Held back: explaining which items move with item features (in What is an item?); then-tests and other designs for response shift are named only; item-level effects in polytomous outcomes are left out, as in the deep dive; life events with real data wait for an IRW table with item-level data around a bereavement.
  • Data: gilbert_meta_20 has a pretest (wave 0, 37 items) and a posttest (wave 1, the same 37 items and 56 more); the table doesn’t say what the trans_math block is beyond its label. gilbert_meta_37 is the endline survey only, and its treat pools two film arms against the control arm. Its std_baseline closely tracks the number right on three knowledge questions asked at baseline (Carpena, 2024, baseline summary statistics), from our check against the replication data.
  • Across the IRW: deepdives/invariance-experience/compute.R is run by hand (it needs a Redivis token); the page reads its results.rds. The pilot took about three minutes for five tables.
  • Solutions: held back in solutions/invariance-experience.qmd (not published). Problem 6 is open.