Linking and equating

Module: Using calibrated items
Tranche: core (a first course)
Before this: From the 1PL to the 4PL
Builds toward: none
Picks up threads: Only θ − b is identified; the scale has no origin (from The Rasch model); The sum score depends on which items were taken (from Where CTT breaks: toward item response models); Whether a scale has an equal unit changes group comparisons (from What is measurement?)
IRW tables: cdm_timss11, pirlsmissing_sirt
From EDUC 252: slides c6; problem sets PS6#4; code c6/equate-irt.R, c6/equate-obsscore.R, c6/concurrent.R, ps6/equating.R, ps6/equating_simstudy.R
Status: draft
Authorship: Written largely by Claude (Anthropic), from Ben Domingue’s EDUC 252 materials and under Ben’s direction. Ben reviews each lesson before it is marked done.
Comments: open an issue on GitHub about this lesson.

What this is for

In 2011, Austrian fourth graders sitting TIMSS each took one of 14 mathematics booklets. Students given booklet 13 averaged 15.0 of its 26 items; those given booklet 11 averaged 11.6. Is the booklet-13 group better at mathematics, or is booklet 13 easier? The booklets were handed out at random, so it is the booklet. The Rasch model said that \(\hat\theta\) earns its keep when respondents take different items, and this lesson collects. Putting scores from different forms on one scale is linking; its strictest form, equating, makes them interchangeable. Whether either is possible is settled mostly by the design. After that, everything rests on an assumption about the items the forms share, which the TIMSS booklets let us check: it is off by a small amount per link, and fourteen links add it up.

Goals

By the end of this lesson you should be able to:

  • say what equating requires (symmetry, equity, population invariance) and which data-collection designs make it possible;
  • equate two forms by mean, linear and equipercentile equating, and say what each assumes about the forms;
  • put two IRT calibrations on one scale with mean-mean, mean-sigma, Haebara or Stocking–Lord linking, or with one concurrent fit, and convert raw scores by IRT true-score equating;
  • say what linking assumes about the common items, and check it in data (position effects, items that behave differently across groups);
  • tell equating from the looser kinds of linking: vertical scaling and concordance.

Core ideas

Two scenarios, one problem

Recall from Where CTT breaks: the sum score depends on which items were taken. The same 664 students averaged 9.6 on an easy half of a test and 5.7 on a hard half, and no single constant converted one half’s score into the other’s.

A testing program writes a new form each year to the same blueprint and needs a score of 30 this year to mean what 30 meant last year. That is equating: afterwards, which form a respondent took shouldn’t matter. A grade 3 test and a grade 4 test that share some items are built to different specifications; their scores can go on one scale, to talk about growth, but can’t be made interchangeable. That is vertical scaling, one kind of the broader family called linking (Linn, 1993; Dorans, Pommerich & Holland, 2007; Kolen & Brennan, 2014). What growth on such a scale means is a question for Scale properties. Looser still, a concordance pairs scores with the same percentile rank on tests built by different programs, such as the SAT and the ACT, without claiming the scores are interchangeable (Dorans, 2004). This lesson is about equating, and about linking forms through the items they share.

What equating asks for

What would make a converted score as good as the original? The literature asks for three properties (Kolen & Brennan, 2014, ch. 1; Dorans & Holland, 2000):

  • Symmetry. Converting form X to form Y and back again returns the score we started with.
  • Equity. A respondent at any level of the construct should be indifferent to which form they get (Lord, 1980).
  • Population invariance. The conversion shouldn’t depend on the group it was computed in; Dorans and Holland read one that does as a sign that the tests shouldn’t be equated.

Why not regression? To predict a respondent’s score on Y from their score on X, regression is the right tool. But predictions shrink toward the mean, so predicting Y from X doesn’t invert predicting X from Y, and symmetry fails.

Try it: regression is not equating

Three hundred respondents take both forms (a single-group design). Form X has mean 12 and SD 4.5, form Y mean 15 and SD 4.3.

The two regression lines disagree, each pulling toward the mean from its own side. Linear equating sits between them, matching means and SDs and ignoring the correlation. Only at a correlation of 1 do the three lines meet.

Design comes first. Equating is impossible unless the data were collected so that a solution exists. Three designs do that (Kolen & Brennan, 2014, ch. 1):

  • Random groups. Forms are handed out at random, so any difference in scores belongs to the forms. No common items are needed.
  • Single group, counterbalanced. Everyone takes both forms, half in each order, so that order effects cancel.
  • Common-item nonequivalent groups (NEAT: nonequivalent groups with an anchor test). The groups may differ (this year’s cohort and last year’s), and common items, the anchor, carry the link, so everything depends on the anchor behaving the same way in both groups.

TIMSS combines two of these: booklets are spiralled through classrooms (random groups), and each booklet also shares a block of items with the next (common items).

Observed-score equating

With random groups, the two score distributions estimate how one population would score on each form, so matching the distributions matches the forms.

  • Mean equating adds a constant: \(y = x - \mu_X + \mu_Y\). It assumes the forms differ only in overall difficulty.
  • Linear equating matches means and SDs: \(y = \mu_Y + \frac{\sigma_Y}{\sigma_X}(x - \mu_X)\), the black line in the widget above. It allows the forms to differ in spread too, and assumes the distributions have similar shapes.
  • Equipercentile equating sends a score on X to the score on Y with the same percentile rank (Kolen & Brennan, 2014, ch. 2). It doesn’t assume similar shapes, which matters when they differ, for instance when one form has a ceiling. Small samples give bumpy distributions, so they are usually smoothed first, for instance by log-linear models that keep their first few moments (Holland & Thayer, 2000).

The equate package (Albano, 2016) does all three.

Try it: equipercentile by hand

Two 25-item Rasch forms, equivalent groups (ability normal, mean 0, SD 1). Each form’s items span three logits around the mean difficulty you set.

The right panel draws the whole conversion, your score marked, against the identity line (dashed). As the widget loads, form Y is two logits easier and many respondents are near its top. Linear equating sends 20 on X to 27.0, more than form Y has; equipercentile equating sends it to 24.7, and the curve bends to stay under 25. Equal difficulties put the curve on the identity line.

IRT linking: through common items, or in one fit

Recall from The Rasch model: only \(\theta - b\) is identified, so the scale has no origin, and mirt sets one by fixing the mean ability at 0. From the 1PL to the 4PL added the unit: the 2PL sees only \(a(\theta - b)\), so software fixes the SD of ability at 1.

Calibrate form X in one group and form Y in another, and each fit sets its origin and unit by its own group. The same item then gets parameters in the two fits related by a linear map: with \(\theta_Y = A\,\theta_X + B\), \[b_Y = A\,b_X + B, \qquad a_Y = a_X / A,\] which leaves every \(a(\theta - b)\), and so every probability, unchanged. Under the Rasch model \(A = 1\) and only the shift \(B\) is needed. The common items give two sets of estimates of the same parameters, from which to find \(A\) and \(B\):

  • Mean-mean (Loyd & Hoover, 1980): \(A = \bar a_X / \bar a_Y\), then \(B = \bar b_Y - A\,\bar b_X\).
  • Mean-sigma (Marco, 1977): \(A = s(b_Y)/s(b_X)\), then \(B\) as before.
  • Haebara (1980): choose \(A\) and \(B\) to bring each common item’s transformed curve close to its curve in the other calibration, over a grid of \(\theta\).
  • Stocking–Lord (Stocking & Lord, 1983): the same for the sum of the curves, the common items’ test characteristic curve (TCC).

Recall from What is measurement?: an interval scale allows exactly the linear relabellings \(\beta_0 + \beta_1 x\), and whether a scale has an equal unit changes group comparisons.

What does each approach assume about the scale? Equipercentile equating allows any order-preserving map: it treats sum scores as ordinal. IRT linking allows only linear maps, the interval scale’s family: it takes \(\theta\) to have an equal unit that the two fits labelled differently. So we link on \(\theta\) and convert to raw scores afterwards if needed. In IRT true-score equating (Lord, 1980), a raw score \(x\) on form X is sent to the \(\theta\) at which X’s TCC equals \(x\), and then to Y’s TCC at that \(\theta\).

Try it: Stocking–Lord by hand

Ten common items. The second calibration’s group has mean 0.5 and SD 1.2 on the first group’s scale, so the right answer is \(A = 1.2\), \(B = 0.5\). Both calibrations carry estimation error, with the SD you set.

The small panels show nine of the items: each one’s second-calibration curve before (light) and after (blue) your \(A\) and \(B\), and its first-calibration curve (dashed), the target. Move one slider and every item moves at once: a link transforms the whole scale, not one item. Close the shaded gap and you have done Stocking–Lord by eye. With no estimation error all three methods recover 1.2 and 0.5. With error they part: at the default SD of 0.15, Stocking–Lord gives 1.23 and 0.55, mean-sigma 1.28 and 0.57. Mean-sigma rests on the difficulties’ spread, which estimation error inflates.

Or skip the linking. IRT handles responses missing by design, so we can stack every form’s data, leave each respondent’s unseen items empty and fit once: concurrent calibration. In the TIMSS booklets most pairs of items are never answered by the same student (Item response data and the IRW counted them), and one fit copes. In simulations it generally gave smaller errors than separate calibration and linking, though not always (Kim & Cohen, 1998; Hanson & Béguin, 2002). Both routes assume a common item’s parameters are the same in every group; a concurrent fit builds that in, while separate calibrations let you compare. A concurrent fit assumes the groups come from one population or differ only in their ability distributions; separate calibrations let you see how the common items behaved.

Assumptions: two calibrations of the same \(n\) common items under the 2PL, with estimates \((\hat a_{iX}, \hat b_{iX})\) and \((\hat a_{iY}, \hat b_{iY})\); the true parameters satisfy \(b_{iY} = A b_{iX} + B\) and \(a_{iY} = a_{iX}/A\); a weight function \(w(\theta)\) on Y’s scale over a grid \(\theta_1, \dots, \theta_Q\) (for instance a normal density).

Moment methods. Mean-sigma sets \(A = s(\hat b_Y)/s(\hat b_X)\). Every item counts equally, however well it is estimated. An item with a slope near zero has a difficulty that is a ratio with a small denominator (\(b = -d/a\) in mirt), so its \(\hat b\) can be far out and dominate both SDs. Mean-mean takes \(A\) from the slopes, which avoids that particular problem, but it still weights every item equally.

Characteristic-curve methods. Stocking–Lord minimizes \[F(A, B) = \sum_{q=1}^{Q} w(\theta_q)\, D_q^2, \qquad D_q = \sum_i P_i(\theta_q;\, \hat a_{iY}, \hat b_{iY}) - \sum_i P_i\big(\theta_q;\, \hat a_{iX}/A,\, A \hat b_{iX} + B\big).\] Write \(P^*_{iq}\) for the transformed curve, \(\text{logit}^{-1}\big((\hat a_{iX}/A)(\theta_q - A\hat b_{iX} - B)\big)\). Since \(\partial P^*_{iq}/\partial B = -(\hat a_{iX}/A)\, P^*_{iq}(1 - P^*_{iq})\), \[\frac{\partial F}{\partial B} = 2\sum_q w(\theta_q)\, D_q \sum_i \frac{\hat a_{iX}}{A}\, P^*_{iq}(1 - P^*_{iq}),\] and \(\partial F/\partial A\) has the same form, with a further factor \((\theta_q - B)/A\) inside the sum over items. An item enters in proportion to \(a\,P(1-P)\), its information (Information, precision, and short forms), at the \(\theta\) values that carry weight. A flat item, or one far from the respondents, barely moves the answer, however extreme its \(\hat b\). Haebara’s criterion squares each item’s difference before summing over items, so each item’s curve must match on its own; Stocking–Lord lets errors in different items cancel.

What follows. The characteristic-curve methods match what the link is for, expected scores where the respondents are, and are less sensitive to poorly estimated items. Moment methods have closed forms, and a large gap between the two families is itself a sign that some common items need a look. The same idea returns when choosing anchor items in Differential item functioning and when adding items to a bank in Item banks and adaptive testing.

How linking breaks

Every method above assumes that a common item is the same item in both places. What can make it not so? Its position or context: an item late in a long test, or next to different items, can be harder there (Leary & Dorans, 1985; Debeer & Janssen, 2013). Drift: an item reused over years can become easier as it is taught to or exposed. Translation: an item in another language is, strictly, another item. Michaelides (2010) reviews what such items do to linking. Each is a form of differential item functioning, an item working differently in two groups matched on the construct (Differential item functioning). And links chain: link C to B and B to A, and each link’s error, and any bias it carries, is passed along. Haberman (2009) develops a regression that links many separately calibrated forms at once rather than pair by pair.

Simulate

The code below gives each of 600 respondents 15 of a bank of 50 Rasch items, in three designs from EDUC 252’s problem set 6: a chain of 8 forms, each overlapping the next by 10 items; 5 forms linked by 5 items, looping back to the first; and 15 random items each. A fourth design repeats the chain but makes an item 0.3 logits harder in the last third of a form. Each design is fit once, concurrently. The first run takes 20–30 seconds while R and mirt load, and the fits a little longer.

With the seed as written, the three designs recover the bank about equally well (correlations 0.96 to 0.97; root mean squared errors 0.16 to 0.19). With the position effect the correlation drops to 0.90, and the errors run along the chain, from 0.42 too easy for the first five items to 0.65 too hard for the last five: the fit has no term for position, so the common items carry it from form to form. At np <- 150 every correlation drops (0.81 to 0.87). At 3,000, in your own R session, the three designs reach 0.99, and the position effect still spreads the errors from −0.33 to 0.39: more data don’t remove a bias.

Download this code to run it in your own R session.

With real data

TIMSS 2011: random groups and a ring of booklets

Our main example is cdm_timss11, 4,668 Austrian fourth graders’ mathematics responses in TIMSS 2011, released with the CDM package by George and Robitzsch (2014). Its job is to show both kinds of design on one table and to test the assumption linking rests on; Item response data and the IRW used it for missingness by design. TIMSS 2011 put its 14 mathematics blocks into 14 booklets, each block in two: M02 is the second mathematics block of booklet 1 and the first of booklet 2, and booklet 14 carries M01 round to booklet 1 (Mullis, Martin, Ruddock, O’Sullivan & Preuschoff, 2009, Exhibit 11). Booklets were distributed so that their groups are approximately equivalent (the same source, ch. 4). A response is 1 when correct; parts of one question (IDs ending A, B, C) are treated as separate items. Download the code to run every chunk yourself.

Code 
library(mirt)
# Each IRW table has a landing page (itemresponsewarehouse.org/tables/<name>/)
# with a CSV download. This link is pinned to one version of the data.
timss_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v59_0.cdm_timss11/rows?format=csv"
tm <- read.csv(timss_url)
# resp is 1 for a correct answer. Each student took one of 14 booklets.
long2wide <- function(df) {
  wide <- tapply(df$resp, list(df$id, df$item), function(x) x[1])
  as.data.frame(wide[, order(colnames(wide))])
}
booklet <- tapply(tm$booklet, tm$id, function(x) x[1])   # one booklet per student
score   <- tapply(tm$resp, tm$id, sum)                    # sum score on that booklet
ib <- table(tm$item, tm$booklet) > 0                      # item x booklet: is the item in it?
c(students = length(booklet), items = nrow(ib), booklets_per_item = unique(rowSums(ib)))
         students             items booklets_per_item 
             4668               174                 2
Code 
# Did every student answer every item in their booklet?
all(table(tm$id)[names(booklet)] == colSums(ib)[as.character(booklet)])
[1] TRUE
Code 
data.frame(booklet = 1:14, students = as.vector(table(booklet)), items = colSums(ib),
           mean_score = round(as.vector(tapply(score, booklet, mean)), 1))
booklet students items mean_score
1 341 21 9.6
2 334 26 13.9
3 335 26 14.0
4 333 23 11.6
5 331 25 15.3
6 330 27 15.1
7 337 25 12.5
8 334 26 12.3
9 338 25 12.0
10 340 25 11.8
11 321 26 11.6
12 336 26 14.0
13 325 26 15.0
14 333 21 9.9
Code 
# Items shared by each booklet and the next (booklet 14 and booklet 1 close the ring)
sapply(1:14, function(k) sum(ib[, k] & ib[, k %% 14 + 1]))
 [1] 12 14 12 11 14 13 12 14 11 14 12 14 12  9

Each booklet shares 9 to 14 items with the next.

Random groups. Booklets 11 and 13 share no items, but their groups are equivalent, so we can equate their score distributions.

Code 
library(equate)
# Booklets 11 and 13 share no items. Their groups are randomly equivalent, so
# observed-score equating needs no common items: match the score distributions.
x <- score[booklet == 11]; y <- score[booklet == 13]
round(c(n_11 = length(x), mean_11 = mean(x), sd_11 = sd(x),
        n_13 = length(y), mean_13 = mean(y), sd_13 = sd(y)), 1)
   n_11 mean_11   sd_11    n_13 mean_13   sd_13 
  321.0    11.6     4.7   325.0    15.0     4.3
Code 
fx <- freqtab(x, scales = 0:26); fy <- freqtab(y, scales = 0:26)
eq_mean   <- equate(fx, fy, type = "mean")
eq_linear <- equate(fx, fy, type = "linear")
eq_equip  <- equate(fx, fy, type = "equipercentile")
# Equipercentile after log-linear presmoothing that keeps each distribution's
# first three moments (Holland & Thayer, 2000)
eq_smooth <- equate(fx, fy, type = "equipercentile", smoothmethod = "loglinear", degrees = list(3, 3))
conv <- data.frame(score_11 = 0:26, mean = eq_mean$concordance$yx, linear = eq_linear$concordance$yx,
                   equipercentile = eq_equip$concordance$yx, smoothed = eq_smooth$concordance$yx)
round(conv[conv$score_11 %in% c(2, 4, 8, 12, 16, 20, 24), ], 1)
score_11 mean linear equipercentile smoothed
3 2 5.4 6.3 3.5 5.5
5 4 7.4 8.1 7.5 7.7
9 8 11.4 11.7 12.0 11.8
13 12 15.4 15.3 15.6 15.5
17 16 19.4 18.9 18.5 19.0
21 20 23.4 22.6 22.0 22.2
25 24 27.4 26.2 26.5 25.0

A score of 12 on booklet 11 converts to 15.4 by mean equating, 15.3 by linear and 15.5 by smoothed equipercentile equating. At the top they part: 24 converts to 27.4 and 26.2 by the first two, more than booklet 13’s 26 items allow, and to 25.0 by the third.

Through the common items. Now fit all 14 booklets at once (a concurrent 2PL) and convert booklet 11 to 13 by IRT true-score equating. Here the common items, not the equivalence of the groups, tie the booklets together.

Code 
# One concurrent 2PL fit: all 14 booklets stacked, each student's unseen items
# missing by design. The common items put every booklet on one scale.
resp <- long2wide(tm)
fit2 <- mirt(resp, 1, itemtype = "2PL", verbose = FALSE)
cf <- coef(fit2, simplify = TRUE)$items          # mirt's a*theta + d; b = -d/a
tcc <- function(theta, items) sum(plogis(cf[items, "a1"] * theta + cf[items, "d"]))
# IRT true-score equating: find the theta whose expected score on booklet 11 is x,
# then report that theta's expected score on booklet 13.
items_11 <- rownames(ib)[ib[, "11"]]; items_13 <- rownames(ib)[ib[, "13"]]
true_score <- sapply(1:25, function(s) {
  th <- uniroot(function(t) tcc(t, items_11) - s, c(-15, 15))$root
  tcc(th, items_13)
})
compare <- data.frame(score_11 = 1:25, irt_true_score = true_score,
                      equipercentile = conv$smoothed[2:26])
compare$difference <- compare$irt_true_score - compare$equipercentile
round(compare[compare$score_11 %in% c(2, 4, 8, 12, 16, 20, 24), ], 1)
score_11 irt_true_score equipercentile difference
2 2 3.3 5.5 -2.3
4 4 6.4 7.7 -1.3
8 8 11.4 11.8 -0.4
12 12 15.3 15.5 -0.3
16 16 18.9 19.0 -0.1
20 20 22.6 22.2 0.4
24 24 25.4 25.0 0.4
Code 
# Largest gap between the two methods where most students are (scores 6 to 18)
round(max(abs(compare$difference[compare$score_11 %in% 6:18])), 2)
[1] 0.65

A score of 12 converts to 15.3, against 15.5. Two routes with different assumptions agree within 0.65 of a point from scores 6 to 18, where most students are, and differ more low down, where few are (by 2.3 at a score of 2).

The ring. The design also lets us check the common items. Calibrate each booklet on its own (Rasch), so each fit centres its own group at 0. With equivalent groups those origins should agree, and mean-mean linking from each booklet to the next should find shifts near zero. Chain the links from booklet 1 through 14 and back to 1.

Code 
# Calibrate each booklet on its own (Rasch; mirt centres each group's abilities at 0),
# then link booklet k+1 to booklet k by mean-mean linking through their common block:
# the shift is the mean difficulty of the common items in k minus that in k+1.
calib <- lapply(1:14, function(k) {
  m <- mirt(long2wide(tm[tm$booklet == k, ]), 1, itemtype = "Rasch", verbose = FALSE)
  d <- coef(m, simplify = TRUE)$items[, "d"]
  -d                                               # Rasch: b = -d
})
shift <- sapply(1:14, function(k) {
  k2 <- k %% 14 + 1
  common <- intersect(names(calib[[k]]), names(calib[[k2]]))
  mean(calib[[k]][common]) - mean(calib[[k2]][common])
})
round(shift, 2)
 [1]  0.20  0.15 -0.06  0.39  0.09  0.22 -0.11  0.00  0.03  0.08  0.02  0.20
[13] -0.01 -0.13
Code 
# Go round the ring, 1 -> 2 -> ... -> 14 -> 1: booklet 1 should end where it started.
round(sum(shift), 2)    # the gap after 14 links
[1] 1.08
Code 
sum(shift > 0)          # how many links shift the same way
[1] 9

The ring misses by 1.08 logits. Why? Each link’s common block is the second mathematics block of one booklet and the first of the next, so if an item is harder when it comes second, every link picks up the same bias. A Rasch model with a term for the second block, fit as a mixed logistic regression (Bates, Mächler, Bolker & Walker, 2015), estimates it directly:

Code 
library(lme4)
# In booklet k the block shared with booklet k-1 comes first and the block shared
# with booklet k+1 second (TIMSS 2011 Assessment Frameworks, Exhibit 11). Mark each
# response by whether its block is the booklet's second mathematics block.
first_in <- apply(ib, 1, function(r) { w <- which(r); if (all(w == c(1, 14))) 1 else max(w) })
tm$second <- as.integer(tm$booklet != first_in[tm$item])
# Odd booklets put mathematics in part 1 (block positions 1-2), even booklets in
# part 2 (positions 3-4), after the break.
tm$position <- ifelse(tm$booklet %% 2 == 1, 1, 3) + tm$second
table(position = tm$position)
position
    1     2     3     4 
29557 28260 28400 29766
Code 
# A Rasch model as a mixed logistic regression: one fixed effect per item, a random
# effect per student, and a shift for the second block (Debeer & Janssen, 2013).
pos1 <- glmer(resp ~ 0 + item + second + (1 | id), data = tm, family = binomial, nAGQ = 0)
round(coef(summary(pos1))["second", 1:2], 3)
  Estimate Std. Error 
    -0.077      0.014
Code 
# The same with each position separately. Each item sits at positions 1 and 4, or 2
# and 3, so the model compares 4 with 1 and 2 with 3.
pos2 <- glmer(resp ~ 0 + item + factor(position) + (1 | id), data = tm, family = binomial, nAGQ = 0)
round(coef(summary(pos2))[c("factor(position)4", "factor(position)2"), 1:2], 3)
                  Estimate Std. Error
factor(position)4   -0.120      0.034
factor(position)2   -0.035      0.034

An item is 0.077 logits harder in the second block (SE 0.014), and 14 × 0.077 = 1.08: the position effect accounts for the gap. Most of it is at the end. An item costs 0.12 logits more in position 4, the last block of the test, than in position 1 (SE 0.034); between positions 2 and 3, either side of the break, the difference is 0.035, within its error. Per item the effect is small; fourteen links make it large.

What does each route show for booklets like these? Both assume a common item has one set of parameters wherever it appears. The concurrent fit builds that in, giving each item one difficulty averaged over its two positions, and uses all the data at once; separate calibrations show whether the common items agree, and only they could have turned up this position effect. Once it is found, the assumption becomes a term in a model, as in the mixed model above, and the data can speak to it directly.

PIRLS 2011: nonequivalent groups, three languages

Our contrast table is pirlsmissing_sirt: one reading booklet of PIRLS 2011, the PIRLS Reader, taken by fourth graders in Austria, Germany, France and the Netherlands, as released with the sirt package (Robitzsch, 2022). Its job is the opposite design: every item is common, the groups are countries, and the link carries the whole comparison, in three languages (German in Austria and Germany). A response is 1 when correct; responses missing by intention and not reached (9%; Item response data and the IRW looked at them) stay missing rather than counting as wrong. We don’t use the student weights, so nothing below estimates a country’s performance.

Code 
pirls_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v59_0.pirlsmissing_sirt/rows?format=csv"
pr <- read.csv(pirls_url)
# resp is 1 for a correct answer; blank for missing by intention and not reached,
# which we leave missing (not wrong). wt is the student weight, unused here.
country <- tapply(pr$country, pr$id, function(x) x[1])
pw <- long2wide(pr)
ncol(pw)                         # items
[1] 35
Code 
round(mean(is.na(pr$resp)), 2)   # share of responses missing
[1] 0.09
Code 
data.frame(students = as.vector(table(country)),
           prop_correct = round(as.vector(tapply(rowMeans(pw, na.rm = TRUE), country, mean)), 2),
           row.names = names(table(country)))
students prop_correct
AUT 968 0.61
DEU 809 0.68
FRA 901 0.61
NLD 802 0.66

We fit a 2PL in each country and link the other three to Austria by all four methods. Each fit sets its country’s mean at 0 and SD at 1, so \(B\) is the country’s mean in Austrian SD units and \(A\) its SD relative to Austria’s.

Code 
# A separate 2PL for each country; each fit sets its own country's mean to 0 and SD to 1.
pfit <- lapply(c(AUT = "AUT", DEU = "DEU", FRA = "FRA", NLD = "NLD"),
               function(k) mirt(pw[country == k, ], 1, itemtype = "2PL", verbose = FALSE))
ab <- lapply(pfit, function(m) coef(m, IRTpars = TRUE, simplify = TRUE)$items[, c("a", "b")])
# Put form X (a country) on the scale of form Y (Austria): theta_Y = A theta_X + B,
# so b -> A b + B and a -> a / A.
grid_th <- seq(-4, 4, length.out = 81); w_th <- dnorm(grid_th)   # quadrature on Austria's scale
icc <- function(a, b) plogis(outer(grid_th, b, "-") * rep(a, each = length(grid_th)))
link <- function(X, Y, method) {
  aX <- X[, "a"]; bX <- X[, "b"]; aY <- Y[, "a"]; bY <- Y[, "b"]
  if (method == "mean-mean")  { A <- mean(aX) / mean(aY); return(c(A = A, B = mean(bY) - A * mean(bX))) }
  if (method == "mean-sigma") { A <- sd(bY) / sd(bX);     return(c(A = A, B = mean(bY) - A * mean(bX))) }
  PY <- icc(aY, bY)
  loss <- function(p) {
    PX <- icc(aX / p[1], p[1] * bX + p[2])
    if (method == "Haebara") sum(w_th * rowSums((PY - PX)^2))       # item by item
    else sum(w_th * (rowSums(PY) - rowSums(PX))^2)                   # Stocking-Lord: the sums
  }
  setNames(optim(c(1, 0), loss)$par, c("A", "B"))
}
methods <- c("mean-mean", "mean-sigma", "Haebara", "Stocking-Lord")
# B is each country's mean in Austrian SD units; A is its SD relative to Austria's.
B <- sapply(c("DEU", "FRA", "NLD"), function(k) sapply(methods, function(m) link(ab[[k]], ab$AUT, m)["B"]))
A <- sapply(c("DEU", "FRA", "NLD"), function(k) sapply(methods, function(m) link(ab[[k]], ab$AUT, m)["A"]))
rownames(B) <- rownames(A) <- methods
round(B, 2)
               DEU   FRA  NLD
mean-mean     0.45  0.04 0.32
mean-sigma    0.44  0.09 0.26
Haebara       0.34 -0.01 0.26
Stocking-Lord 0.36 -0.02 0.27
Code 
round(A, 2)
               DEU  FRA  NLD
mean-mean     0.98 0.98 0.81
mean-sigma    0.97 1.09 0.73
Haebara       0.98 0.97 0.81
Stocking-Lord 0.99 1.00 0.80
Code 
# Cross-check with the equateIRT package (Battauz, 2015): Stocking-Lord for Germany
library(equateIRT)
mods <- modIRT(coef = lapply(ab, function(x) cbind(b = x[, "b"], a = x[, "a"])),
               names = names(ab), ltparam = FALSE, display = FALSE)
round(unlist(direc(mods = mods, which = c("DEU", "AUT"), method = "Stocking-Lord")[c("A", "B")]), 2)
   A    B 
0.99 0.36

France sits within 0.09 of Austria by every method. For Germany the method matters: 0.44 and 0.45 by the moment methods, 0.34 and 0.36 by Haebara and Stocking–Lord, a tenth of an SD apart on the same responses. The Netherlands is 0.26 to 0.32 above Austria, with an SD 0.73 to 0.81 of Austria’s. (The equateIRT package, Battauz, 2015, gives the same Stocking–Lord constants for Germany.) Do the common items behave the same in each country?

Code 
# After Stocking-Lord linking, how far does each item's difficulty in the Netherlands
# sit from its difficulty in Austria? Linking assumes these differences are noise.
item_gap <- function(k) {
  p <- link(ab[[k]], ab$AUT, "Stocking-Lord")
  (p["A"] * ab[[k]][, "b"] + p["B"]) - ab$AUT[, "b"]
}
gap_nld <- item_gap("NLD")
round(range(gap_nld), 2)
[1] -1.20  0.92
Code 
sapply(c(DEU = "DEU", FRA = "FRA", NLD = "NLD"), function(k) sum(abs(item_gap(k)) > 0.5))
DEU FRA NLD 
  6  10  10
Code 
# Drop the Dutch items more than 0.5 logits out and link again on the rest
keep <- names(gap_nld)[abs(gap_nld) <= 0.5]
round(link(ab$NLD[keep, ], ab$AUT[keep, ], "Stocking-Lord"), 2)
   A    B 
0.85 0.27

After Stocking–Lord linking, Dutch difficulties differ from Austrian ones by −1.20 to 0.92 logits. Ten of the 35 items are more than half a logit out for the Netherlands, ten for France, and six for Germany, which shares Austria’s language. Dropping the ten Dutch items and linking again leaves the Dutch mean at 0.27. Here the method matters more than the trimming: the item differences largely cancel. They are differential item functioning all the same, and whether the countries can be compared on this booklet is the question Differential item functioning asks.

Problems

  1. Derivation. Show that under the 2PL, with \(\theta_Y = A\theta_X + B\), the maps \(b_Y = Ab_X + B\) and \(a_Y = a_X/A\) leave every probability unchanged, and derive the mean-mean constants. Then show that predicting Y from X by regression, and X back from that by the other regression, returns the starting score only if the correlation is 1.
  2. A devious split (from EDUC 252). Take TIMSS booklets 11 and 12, which share a block, and keep only booklet 12’s top 70% by sum score. Link 12 to 11 through the common block (mean-mean, Rasch), and equate the sum scores by equipercentile equating as if the groups were random. What happens to each? Which notices the selection?
  3. Judgment. Make the best case for reporting TIMSS results from separate calibrations rather than the concurrent fit. What would you do about the 1.08-logit gap?
  4. Design (EDUC 252, problem set 6). You can test 1,500 respondents, 15 items each, from a 50-item bank. Using the simulation, compare the three designs, then shrink the loop’s links to two items, or one. When does the design start to matter?
  5. Real data. In the PIRLS data, which would you trust more, Germany’s link to Austria or France’s? What would you check first?
  6. Challenge (open). A testing program moves a block of common items from the start of this year’s form to the end of next year’s. Design a study that could tell a position effect from a change in the population. What would you randomize, and what would it cost? I don’t know of a design that settles this cheaply.

Stuck, or curious about something this lesson doesn’t cover? Ask Claude about this lesson opens a new conversation that already knows what the lesson is about; add your question at the end. Claude shows a caution notice on any prompt that arrives through a link; that’s expected.

Going further

  • Kolen, M. J., & Brennan, R. L. (2014). Test equating, scaling, and linking: Methods and practices (3rd ed.). Springer. https://doi.org/10.1007/978-1-4939-0317-7. The standard reference: designs, observed-score methods, smoothing, IRT linking and standard errors.
  • Dorans, N. J., Pommerich, M., & Holland, P. W. (Eds.). (2007). Linking and aligning scores and scales. Springer. https://doi.org/10.1007/978-0-387-49771-6. Where equating sits in the wider family of linkages.
  • Linn, R. L. (1993). Linking results of distinct assessments. Applied Measurement in Education, 6(1), 83–102. https://doi.org/10.1207/s15324818ame0601_5. The kinds of linking, from equating to the loosest.
  • Dorans, N. J. (2004). Equating, concordance, and expectation. Applied Psychological Measurement, 28(4), 227–246. https://doi.org/10.1177/0146621604265031. How to tell whether two tests can be equated or only concorded, with the SAT and the ACT.
  • Dorans, N. J., & Holland, P. W. (2000). Population invariance and the equatability of tests: Basic theory and the linear case. Journal of Educational Measurement, 37(4), 281–306. https://doi.org/10.1111/j.1745-3984.2000.tb01088.x. Measuring whether a conversion depends on the group.
  • Lord, F. M. (1980). Applications of item response theory to practical testing problems. Erlbaum. https://doi.org/10.4324/9780203056615. Equity and IRT true-score equating.
  • Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7(2), 201–210. https://doi.org/10.1177/014662168300700208. The TCC criterion.
  • Haebara, T. (1980). Equating logistic ability scales by a weighted least squares method. Japanese Psychological Research, 22(3), 144–149. https://doi.org/10.4992/psycholres1954.22.144. The item-by-item criterion.
  • Marco, G. L. (1977). Item characteristic curve solutions to three intractable testing problems. Journal of Educational Measurement, 14(2), 139–160. https://doi.org/10.1111/j.1745-3984.1977.tb00033.x. Mean-sigma linking.
  • Loyd, B. H., & Hoover, H. D. (1980). Vertical equating using the Rasch model. Journal of Educational Measurement, 17(3), 179–193. https://doi.org/10.1111/j.1745-3984.1980.tb00825.x. Mean-mean linking, and an early vertical scale.
  • Kim, S.-H., & Cohen, A. S. (1998). A comparison of linking and concurrent calibration under item response theory. Applied Psychological Measurement, 22(2), 131–143. https://doi.org/10.1177/01466216980222003. Separate against concurrent, by simulation.
  • Hanson, B. A., & Béguin, A. A. (2002). Obtaining a common scale for item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26(1), 3–24. https://doi.org/10.1177/0146621602026001001. The same comparison, with all four linking methods.
  • Haberman, S. J. (2009). Linking parameter estimates derived from an item response model through separate calibrations. ETS Research Report Series, 2009(2). https://doi.org/10.1002/j.2333-8504.2009.tb02197.x. Linking many forms at once.
  • Leary, L. F., & Dorans, N. J. (1985). Implications for altering the context in which test items appear: A historical perspective on an immediate concern. Review of Educational Research, 55(3), 387–413. https://doi.org/10.2307/1170392. Context and position effects.
  • Debeer, D., & Janssen, R. (2013). Modeling item-position effects within an IRT framework. Journal of Educational Measurement, 50(2), 164–185. https://doi.org/10.1111/jedm.12009. Position effects as part of the model, which Explanatory item response models takes up.
  • Michaelides, M. P. (2010). A review of the effects on IRT item parameter estimates with a focus on misbehaving common items in test equating. Frontiers in Psychology, 1, 167. https://doi.org/10.3389/fpsyg.2010.00167. What happens when common items change.
  • Holland, P. W., & Thayer, D. T. (2000). Univariate and bivariate loglinear models for discrete test score distributions. Journal of Educational and Behavioral Statistics, 25(2), 133–183. https://doi.org/10.3102/10769986025002133. The presmoothing used above.
  • von Davier, A. A., Holland, P. W., & Thayer, D. T. (2004). The kernel method of test equating. Springer. https://doi.org/10.1007/b97446. A framework that unifies the observed-score methods, with standard errors.
  • González, J., & Wiberg, M. (2017). Applying test equating methods using R. Springer. https://doi.org/10.1007/978-3-319-51824-4. The methods of this lesson, in R.
  • Albano, A. D. (2016). equate: An R package for observed-score linking and equating. Journal of Statistical Software, 74(8). https://doi.org/10.18637/jss.v074.i08. The package used for observed-score equating.
  • Battauz, M. (2015). equateIRT: An R package for IRT test equating. Journal of Statistical Software, 68(7). https://doi.org/10.18637/jss.v068.i07. IRT linking methods, including chains of forms.
  • Mullis, I. V. S., Martin, M. O., Ruddock, G. J., O’Sullivan, C. Y., & Preuschoff, C. (2009). TIMSS 2011 assessment frameworks. TIMSS & PIRLS International Study Center, Boston College. https://timssandpirls.bc.edu/timss2011/downloads/TIMSS2011_Frameworks.pdf. Chapter 4 sets out the booklet design.
  • Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1). https://doi.org/10.18637/jss.v067.i01. The package for the position model.
  • Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6). https://doi.org/10.18637/jss.v048.i06. The package for every IRT fit here.
  • van der Linden, W. J., & Barrett, M. D. (2017). Parameter linking. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 3. Applications (pp. 21–46). Chapman and Hall/CRC. https://doi.org/10.1201/9781315117430. A handbook chapter on linking item response model parameters across calibrations.
  • van der Linden, W. J. (2017). IRT observed-score equating. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 3. Applications (pp. 143–164). Chapman and Hall/CRC. https://doi.org/10.1201/9781315117430. A handbook chapter on observed-score equating with item response models.

Data sources

Data from the Item Response Warehouse (each table name links to its IRW page); references from IRW biblio.

For instructors

  • Source: EDUC 252 class 6 (slides c6, 33–47), problem set 6 (#4), and the code in ben-domingue/252: c6/equate-obsscore.R, c6/equate-irt.R, c6/concurrent.R, ps6/equating.R, ps6/equating_simstudy.R.
  • Timing: the core ideas and widgets take about 60 minutes; the TIMSS and PIRLS analysis works as a 30-minute lab. Locally, the position model takes about a minute and everything else seconds.
  • Changed from 252: the slides and problem set equate ENEM booklets; this lesson uses TIMSS 2011 instead (Ben, 09-24), whose ring of booklets teaches random groups and common items on one table and turns up a real position effect. PIRLS 2011 across four countries is new, as the nonequivalent-groups contrast. The simulation adds a fourth design with a position effect.
  • Held back: vertical scales and growth (Scale properties); calibrating new items onto a bank’s scale (Item banks and adaptive testing); position effects as an explanatory model (Explanatory item response models); DIF in the anchor (Differential item functioning). Kernel equating, equating standard errors and linking across languages (alignment) are not covered.
  • Solutions: held back in solutions/equating.qmd (not published).