Item banks and adaptive testing

Module: Using calibrated items
Tranche: extension (beyond a first course)
Before this: Information, precision, and short forms · Estimating abilities: MLE and EAP · Models for polytomous responses
Builds toward: none
Picks up threads: Misfit as overfit or underfit (outfit and infit) (from The Rasch model); What we want from a measure: insensitive to nonfocal attributes, calibrated, precise quickly (from What is measurement?); EAP estimates shrink toward the mean (from Estimating abilities: MLE and EAP); No maximum-likelihood estimate for all-right or all-wrong patterns (from Estimating abilities: MLE and EAP)
IRW tables: choi_2026_cmsce_2020_1, promis1wave1_depression
From EDUC 252: slides c6, c10
Status: draft
Authorship: Written largely by Claude (Anthropic), from Ben Domingue’s EDUC 252 materials and under Ben’s direction. Ben reviews each lesson before it is marked done.
Comments: open an issue on GitHub about this lesson.

What this is for

A collection of calibrated items is an item bank, and its most demanding use is computerized adaptive testing (CAT): each item is chosen after the last answer, to suit the respondent. Information, precision, and short forms picked the most informative items for one point on the scale; a CAT does that again after every response, at the respondent’s current estimate. This lesson builds the loop from information and the EAP, asks what stopping rules buy and what choosing by information costs, and runs adaptive tests after the fact on a medical examination and the PROMIS depression items. A CAT is only as good as the bank is where the respondent sits.

Recall from What is measurement?: one of the things we want from a measure is that it give precise information quickly, with few items. Adaptive testing is the most direct attempt at “quickly”.

Goals

By the end of this lesson you should be able to:

  • say what an item bank holds and what keeping one up involves;
  • run the adaptive loop: start, choose by information, score by EAP, stop;
  • compare fixed-length, standard-error and classification stopping rules;
  • explain what maximum information does to item exposure, and what the fixes cost;
  • run a post-hoc CAT on real responses and read it as a statement about the bank.

Core ideas

An item bank

What does a bank store? At the least, each item and its calibrated parameters, fixed, so that a respondent’s \(\theta\) can be estimated from any subset of the items and lands on the bank’s scale (Wainer et al., 2000; van der Linden & Glas, 2010). A working bank also records each item’s content codes, dates, how often it has been given, and the checks it has passed.

Recall from The Rasch model: outfit and infit are near 1 if the model is right; well below 1 an item overfits (it is more predictable than the model expects), well above 1 it underfits. A bank screens its items with statistics like these.

If you’ve done Model fit and out-of-sample prediction, outfit’s spread under a true model shrinks with sample size, so the screening rule should depend on the calibration sample. If you’ve done Differential item functioning, each item also gets a DIF screen, which needs group memberships in the calibration data. If you’ve done Linking and equating, new items join the bank’s scale by calibration alongside old ones acting as anchors. A bank is maintained, not finished: overused items are retired, old items checked for drift, new ones added.

The adaptive loop

Recall from Information, precision, and short forms: a 2PL item’s information is \(a^2P(1-P)\), largest at \(\theta = b\); test information adds across items, and the conditional SEM is \(1/\sqrt{I(\theta)}\).

How do we choose items for someone whose \(\theta\) we don’t yet know? One at a time (Lord, 1980, ch. 10; Weiss, 1982):

  1. Start at the population mean.
  2. Choose the unused item most informative at the current estimate, and give it.
  3. Update the estimate from every response so far.
  4. Stop when a rule says so; otherwise go back to 2.

Step 3 has a problem at the start.

Recall from Estimating abilities: an all-right or all-wrong pattern has no maximum-likelihood estimate; the EAP, the posterior mean under a normal prior, exists for every pattern.

After one item every pattern is all-right or all-wrong, so there is no MLE until the respondent has one of each; the EAP exists throughout, and adaptive testing and Bayesian scoring grew up together (Owen, 1975; Bock & Mislevy, 1982). Adapting is older than computers: Binet and Simon (1904) graded their tasks by the age at which children pass them, and individually given tests still start at a basal level and stop at a ceiling.

Try it: one CAT, step by step

A 100-item 2PL bank and one simulated respondent. Choose their true \(\theta\) and step through the test. In the top panel each dot is one bank item, placed at its difficulty \(b\), at a height equal to its information at the EAP before this step; the orange dot is the item the CAT gives at this step (the most informative one still available). The bottom panel is the posterior after the response.

At the defaults the first four answers are right, and there is no MLE until the fifth, a wrong answer; the EAP climbs meanwhile (0.56, 1.06, 1.41, 1.74) and ends at 1.59, posterior SD 0.34. The first four items have slopes from 2.2 to 2.4, and from step 10 on the slopes are 1.2 to 1.6: maximum information spends the steepest items first, because \(a^2\) multiplies everything else.

Assumptions: known item parameters; local independence; EAP scoring with a normal prior. Write \(p_k(\theta)\) for the posterior after \(k\) items, with mean \(\hat\theta_k\) and variance \(\sigma_k^2\).

What we would like to minimize. If a CAT stops on the posterior SD, a natural choice of next item \(j\) is the one that leaves the smallest posterior variance after its response, averaged over the responses it might get: \[\text{EPV}_j = \sum_{x} \Pr(x_j = x \mid \text{responses so far})\; \text{Var}(\theta \mid \text{responses so far},\, x_j = x),\] where \(\Pr(x_j = x \mid \cdot) = \int \Pr(x_j = x \mid \theta)\,p_k(\theta)\,d\theta\). Owen (1975) selected items with a normal approximation to the posterior; van der Linden (1998) sets out selection criteria that use the posterior itself.

Why maximum information approximates it. Suppose the posterior is close to normal and narrow enough that item \(j\)’s information barely changes across it. Then one more response adds about \(I_j(\hat\theta_k)\) to the posterior’s precision, as a normal likelihood carrying that information would: \[\frac{1}{\sigma_{k+1}^2} \approx \frac{1}{\sigma_k^2} + I_j(\hat\theta_k).\] The right-hand side increases with \(I_j(\hat\theta_k)\), so the item with the most information at the EAP is, approximately, the item that shrinks the posterior most.

When they part. Early in the test the posterior is wide (\(\sigma_0 = 1\)), and a steep item’s information is concentrated in a band much narrower than that: an item with \(a = 2.4\) has lost about 70% of its peak information one logit from its \(b\). What matters then is information averaged over where \(\theta\) might be, \(\int I_j(\theta)\,p_k(\theta)\,d\theta\), which can favour a flatter item near the posterior’s centre. Chang and Ying (1996) argue that information at the estimate is least suited to the start of a test, and propose a global (Kullback–Leibler) information for the early items. As responses accumulate, the posterior narrows, the approximation improves, and the criteria agree.

Stopping rules

When should a CAT stop? Three answers are standard (Weiss & Kingsbury, 1984).

  • Fixed length. Everyone gets, say, 20 items: predictable time, easy content balance, precision that varies along the scale.
  • Target standard error. Stop when the posterior SD reaches, say, 0.3, with a cap on length: equal precision wherever the bank can deliver it, and varying length.
  • Classification. For a pass/fail decision, stop once the responses favour one side of the cut clearly enough (callout below). If you’ve done What does a score mean?, this spends the items near the cut, where classification is uncertain.

Assumptions: known item parameters; local independence; one cut \(\theta_c\) and an indifference region \(\theta_c \pm \delta\) within which either decision is acceptable; tolerated error rates \(\alpha\) (failing someone at \(\theta_c + \delta\)) and \(\beta\) (passing someone at \(\theta_c - \delta\)).

The test. After \(k\) items, compare the likelihood of the responses at the two edges of the region, \[\Lambda_k = \frac{L(\theta_c + \delta \mid x_1, \dots, x_k)}{L(\theta_c - \delta \mid x_1, \dots, x_k)}.\] Wald’s (1945) sequential probability ratio test stops and passes the respondent once \(\Lambda_k \ge (1 - \beta)/\alpha\), stops and fails them once \(\Lambda_k \le \beta/(1 - \alpha)\), and otherwise gives another item. With \(\alpha = \beta = 0.05\) the bounds are 19 and 1/19.

Which items. Every item moves \(\log\Lambda\) by \(\log[P_i(\theta_c+\delta)/P_i(\theta_c-\delta)]\) if right and by the corresponding ratio of \(1 - P_i\) if wrong, so the items that move it most sit near the cut, whatever the current estimate. Choosing by information at the cut is the simple rule; Eggen (1999) compares it with Kullback–Leibler information, and Spray and Reckase (1996) compare the SPRT with a Bayesian sequential rule. Respondents far from the cut are classified in a few items; those near it take the longest, and a cap decides what happens to them. Problem 5 tries it on the CMSCE.

Recall from Estimating abilities: the EAP is pulled toward the prior mean, most where the likelihood is flattest, and its standard error is the posterior SD.

A standard-error rule stops on the posterior SD, and the posterior includes the prior: where the bank is thin, part of the precision it counts is the prior’s, and the estimate that stops has been pulled toward the mean. (Between fixed forms and a CAT sit multistage tests, which adapt between short preassembled modules; Luecht & Nungester, 1998.)

Try it: stopping rules and the shape of the bank

A 150-item bank whose difficulties you place; 25 simulated respondents at each true \(\theta\) from −3 to 3 (target-SE tests stop at 60 items at most).

With 20 items for everyone the RMSE is 0.19 at \(\theta = 0\) and 0.67 at \(-3\): equal length, unequal precision. With a target of 0.3, respondents near the middle stop after about 13 items and those at \(\pm 3\) run to about 57, with an RMSE still 0.52 at \(-3\): the bank has few items out there. Narrow the bank to an SD of 0.3 and the middle gets cheaper (11 items) while respondents at \(\pm 2\) need 53 to 58. The stopping rule decides how the bank’s shape is spent; it can’t change the shape.

What maximum information costs

Choosing the most informative item is efficient, with side effects (Kingsbury & Zara, 1989). Everyone sees the same first item and respondents who answer alike see the same next ones, so a few steep items go to almost everyone; where a test has stakes, such items become known. And information knows nothing about content: a CAT on a medical examination could spend every item on one specialty unless told otherwise.

The fixes trade precision for control (Revuelta & Ponsoda, 1998, compare eight). Randomesque selection picks at random among the few most informative items (Kingsbury & Zara, 1989). Sympson–Hetter gives each selected item a probability of being administered, tuned by simulation to cap its exposure (Hetter & Sympson, 1997; van der Linden, 2003). a-stratification splits the bank by slope and moves from the flattest stratum to the steepest, matching \(b\) to the estimate within each (Chang & Ying, 1999; Chang, 2015).

Try it: who sees which items

Five hundred simulated respondents take a 20-item CAT from a 100-item bank. Each dot is an item, at its difficulty and the share of respondents who saw it.

Under maximum information one item goes to everyone, 17 go to more than half, and 38 of the 100 are never used; the most exposed are steep items near the middle. Randomesque selection among the top five brings the most-seen item down only to 87%, with 30 unused and no loss of precision. a-stratification spreads the load much further (3 items seen by more than half, 9 unused) at a cost: the RMSE rises from 0.28 to 0.34, because the steep items wait.

Everything rests on the bank’s parameters

A CAT puts every respondent’s different test on one scale, which works only if the bank’s parameters, estimated from particular respondents taking a particular form, hold for other respondents and other combinations of items. If you’ve done Parameter invariance, that lesson tested this property. The ROAR-CAT project checked it before going live (Ma et al., 2025): it calibrated a reading bank in four groups of students, removed six items that functioned differently across them, and then randomly assigned students to random or CAT order. To reach a reliability of 0.9, 75 CAT items did the work of 125 random ones.

A cheaper check uses data already in hand: a post-hoc CAT. Calibrate on some respondents who answered every item, and let a CAT choose items for the others, revealing only the responses it asks for; then compare its estimate with the whole bank’s, respondent by respondent. The unchosen items are missing by design, which is ignorable given the model (Mislevy & Wu, 1996). What it can’t show is whether respondents answer an item the same way inside a CAT as inside the fixed form.

Simulate

The code below gives 200 simulated respondents a CAT from a 200-item 2PL bank (EAP, maximum information, stop at a posterior SD of 0.3 or 60 items), and compares it with a fixed form of the 20 items most informative at \(\theta = 0\). It is base R and runs in a second or two.

With the seed as written, the CAT uses 13.65 items on average against the form’s 20, with RMSEs of 0.29 and 0.31. The averages hide a trade: in the middle (\(-0.5\) to \(0.5\)) the fixed form is more precise (0.22 against 0.29), since the CAT stops at 0.3 after about 10 items; above 1.5 the CAT is (0.30 against 0.47), using 26 items. The form spends its items where most respondents are; the CAT spends them where each respondent is. Try se_stop <- 0.2, or b <- rnorm(n_items, 1.5, 0.5), a bank built for a high cut.

Download this code to run it in your own R session.

With real data

A medical examination as a bank

Our main example is choi_2026_cmsce_2020_1: a 2020 sitting of Korea’s Clinical Medical Science Comprehensive Examination (CMSCE) for medical students (Choi et al., 2026, data). Its job is to be a real bank for a post-hoc CAT. The items are five-option multiple choice across seven clinical domains, scored 1 for correct, and Choi et al. (2025) built a self-assessment CAT from six such sittings. Download the code; it needs mirt (Chalmers, 2012), and the CAT engine is a few dozen lines of base R.

Code 
library(mirt)
# Each IRW table has a landing page (itemresponsewarehouse.org/tables/<name>/) with a
# CSV download. This link is pinned to one version of the data.
cm_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse_4:v8_0.choi_2026_cmsce_2020_1/rows?format=csv"
df <- read.csv(cm_url)
# resp is 1 for a correct answer and 0 otherwise. One sitting per respondent: no waves.
cm <- tapply(df$resp, list(df$id, df$item), function(x) x[1])
c(respondents = nrow(cm), items = ncol(cm), missing = sum(is.na(cm)),
  mean_correct = round(mean(cm), 2))
 respondents        items      missing mean_correct 
     5388.00       304.00         0.00         0.68

Every student answered every item. We calibrate a 2PL on a random half and keep the other half to be tested.

Code 
# Calibrate on a random half; the other half is kept back to take the adaptive test.
# mirt writes the 2PL as a*theta + d; our difficulty is b = -d/a (IRTpars = TRUE).
set.seed(2020)
cal <- sort(sample(nrow(cm), nrow(cm) %/% 2))
m_cm <- mirt(as.data.frame(cm[cal, ]), 1, itemtype = "2PL", verbose = FALSE)
pars <- coef(m_cm, IRTpars = TRUE, simplify = TRUE)$items[, c("a", "b")]
# Items with slopes below 0.3 say little about theta anywhere (information a^2/4 at
# most 0.02); they stay out of the bank.
bank_items <- rownames(pars)[pars[, "a"] >= 0.3]
c(calibration = length(cal), items = nrow(pars), low_slope = sum(pars[, "a"] < 0.3),
  of_which_negative = sum(pars[, "a"] < 0), bank = length(bank_items))
      calibration             items         low_slope of_which_negative 
             2694               304                33                11 
             bank 
              271
Code 
round(quantile(pars[bank_items, "b"], c(0, 0.1, 0.5, 0.9, 1)), 2)   # bank difficulties
   0%   10%   50%   90%  100% 
-7.15 -2.73 -1.25  0.20  4.21
Code 
round(quantile(pars[bank_items, "a"], c(0, 0.5, 1)), 2)             # bank slopes
  0%  50% 100% 
0.30 1.05 2.59

Thirty-three items with slopes below 0.3 stay out, leaving 271. Eleven of the 33 have negative slopes, which in a working bank would send them to content review; the scored responses alone can’t say why. The bank’s median difficulty is −1.25, and only a tenth of its items are harder than 0.20: most are easy for these students.

Code 
# A CAT in base R, for any unidimensional model mirt fits. Every quantity lives on a
# grid of nodes: the prior, each item's response probabilities and its information.
nodes <- seq(-6, 6, length.out = 121)
prior <- dnorm(nodes)   # mirt's calibration fixes the mean at 0 and the SD at 1
make_bank <- function(model, items) {
  it <- lapply(items, function(i) extract.item(model, i))
  list(items = items,
       logp = lapply(it, function(e) log(probtrace(e, nodes))),  # nodes x categories
       info = sapply(it, function(e) iteminfo(e, nodes)))         # nodes x items
}
# EAP and posterior SD from the log likelihood on the nodes
eap <- function(ll) {
  w <- exp(ll - max(ll)) * prior
  w <- w / sum(w)
  th <- sum(nodes * w)
  c(theta = th, se = sqrt(sum((nodes - th)^2 * w)))
}
# Each available item's information at theta, interpolated between nodes
info_at <- function(bank, th, avail) {
  k <- findInterval(th, nodes, all.inside = TRUE)
  f <- (th - nodes[k]) / (nodes[k + 1] - nodes[k])
  (1 - f) * bank$info[k, avail] + f * bank$info[k + 1, avail]
}
# One respondent. x holds their responses (0, 1, ...) to the bank items, in bank order.
# Start at the prior mean; give the most informative item at the current EAP (or pick
# at random among the `top` most informative, or at random from the whole bank); update;
# stop at max_items or when the posterior SD reaches se_stop.
run_cat <- function(x, bank, max_items = 20, se_stop = 0, select = "info", top = 1) {
  ll <- numeric(length(nodes))
  used <- integer(0)
  est <- eap(ll)
  while (length(used) < max_items && est[["se"]] > se_stop) {
    avail <- setdiff(seq_along(bank$items), used)
    if (select == "random") {
      j <- avail[sample.int(length(avail), 1)]
    } else {
      best <- avail[order(info_at(bank, est[["theta"]], avail), decreasing = TRUE)]
      best <- best[seq_len(min(top, length(best)))]
      j <- best[sample.int(length(best), 1)]
    }
    ll <- ll + bank$logp[[j]][, x[j] + 1]
    used <- c(used, j)
    est <- eap(ll)
  }
  list(theta = est[["theta"]], se = est[["se"]], n = length(used), used = used)
}
bank_cm <- make_bank(m_cm, bank_items)

The next chunk draws 600 held-out students and computes each one’s full-bank EAP, the answer a CAT tries to reach with fewer items.

Code 
# The respondents who take the CAT: 600 drawn from the held-out half (the precomputed
# run uses all 600; the page runs the first 50 live).
held <- setdiff(seq_len(nrow(cm)), cal)
set.seed(2021)
cat_ids <- sample(held, 600)
X_cm <- cm[cat_ids, bank_items]
# Everyone's full-bank EAP: the target a CAT is trying to reach with fewer items
full_cm <- t(apply(X_cm, 1, function(x) unlist(run_cat(x, bank_cm, max_items = length(bank_items))[1:3])))
Code 
# Where is the bank informative? Test information and CSEM along the scale.
th <- seq(-4, 4, by = 0.01)
I_bank <- rowSums(sapply(bank_items, function(i) iteminfo(extract.item(m_cm, i), th)))
c(peak_at = th[which.max(I_bank)], peak_info = round(max(I_bank), 1),
  csem_at_peak = round(1 / sqrt(max(I_bank)), 2))
     peak_at    peak_info csem_at_peak 
       -1.41        72.20         0.12
Code 
round(sapply(c(-3, -2, -1, 0, 1, 2, 3), function(t) 1 / sqrt(I_bank[abs(th - t) < 1e-9])), 2)
[1] 0.16 0.12 0.12 0.15 0.21 0.31 0.44
Code 
c(items_above_2 = sum(pars[bank_items, "b"] > 2), items_below_minus2 = sum(pars[bank_items, "b"] < -2))
     items_above_2 items_below_minus2 
                 8                 74

The bank’s information peaks at \(\theta = -1.41\) (CSEM 0.12); the CSEM is 0.31 at \(\theta = 2\) and 0.44 at 3. Only 8 items have \(b\) above 2, against 74 below −2, so even the whole bank can’t give a student at \(\theta = 2\) a standard error of 0.3. First, a known answer: a “CAT” allowed the whole bank must reproduce mirt’s full-bank EAP.

Code 
# A known answer first: a "CAT" that gives the whole bank must reproduce mirt's own
# full-test EAP for every respondent (same nodes, same prior). Non-bank items are NA.
pat <- cm[cat_ids, colnames(cm)]
pat[, setdiff(colnames(cm), bank_items)] <- NA
mirt_eap <- fscores(m_cm, method = "EAP", response.pattern = pat, quadpts = 121,
                    theta_lim = c(-6, 6))
c(largest_difference = signif(max(abs(mirt_eap[, "F1"] - full_cm[, "theta"])), 2))
largest_difference 
           1.3e-14

It does, to rounding error. The first items for one student:

Code 
# The first few steps for one respondent: after one or two right answers there is no
# maximum-likelihood estimate, but the EAP moves up and its SD falls.
one <- run_cat(X_cm[1, ], bank_cm, max_items = 6)
steps <- t(sapply(1:6, function(s) {
  ll <- Reduce(`+`, lapply(one$used[1:s], function(j) bank_cm$logp[[j]][, X_cm[1, j] + 1]))
  e <- eap(ll)
  c(item = s, a = round(pars[bank_items[one$used[s]], "a"], 2),
    b = round(pars[bank_items[one$used[s]], "b"], 2), x = X_cm[1, one$used[s]],
    eap = round(e[["theta"]], 2), post_sd = round(e[["se"]], 2))
}))
steps
     item    a     b x   eap post_sd
[1,]    1 2.03 -0.01 0 -0.61    0.79
[2,]    2 2.21 -0.54 0 -1.04    0.68
[3,]    3 2.09 -1.12 0 -1.42    0.62
[4,]    4 2.59 -1.72 1 -1.16    0.50
[5,]    5 2.13 -1.27 0 -1.40    0.45
[6,]    6 2.21 -1.50 0 -1.60    0.42

Three wrong answers, a pattern with no MLE, yet the EAP moves from −0.61 to −1.42; every item chosen has a slope of 2 or more.

Code 
# The first 50 CAT respondents, three rules: 20 items chosen by information, 20 chosen
# at random, and information until the posterior SD reaches 0.3 (at most 100 items).
set.seed(50)
live <- 1:50
rules <- list(info20 = list(max_items = 20), random20 = list(max_items = 20, select = "random"),
              se0.3 = list(max_items = 100, se_stop = 0.3))
live_res <- lapply(rules, function(r) t(sapply(live, function(k)
  unlist(do.call(run_cat, c(list(x = X_cm[k, ], bank = bank_cm), r))[1:3]))))
t(sapply(live_res, function(z) c(
  mean_items = mean(z[, "n"]), r_full = round(cor(z[, "theta"], full_cm[live, "theta"]), 3),
  rmse = round(sqrt(mean((z[, "theta"] - full_cm[live, "theta"])^2)), 2))))
         mean_items r_full rmse
info20         20.0  0.956 0.28
random20       20.0  0.854 0.49
se0.3          30.4  0.967 0.26

For 50 students, 20 items chosen by information correlate 0.96 with the full-bank EAP (RMSE 0.28); 20 at random, 0.85 (0.49). A target SE of 0.3 takes 30 items on average.

Six hundred students, five rules

All 600 students under five rules are precomputed by compute.R, which reuses this page’s code for the data, calibration and engine; its first 50 students are those above.

Code 
res_path <- file.path("..", "deepdives", "item-banks-cat", "results.rds")
cat600 <- if (file.exists(res_path)) readRDS(res_path) else NULL
if (!is.null(cat600)) {
  s <- cat600$summary
  print(data.frame(rule = s$rule, mean_items = round(s$mean_items, 1), median_items = s$median_items,
                   r_full = round(s$r_full, 3), rmse = round(s$rmse, 2),
                   reached_0.3 = round(s$share_reached_0.3, 2),
                   max_exposure = round(s$max_exposure, 2), never_used = s$items_never_used),
        row.names = FALSE)
}
       rule mean_items median_items r_full rmse reached_0.3 max_exposure
   random20       20.0           20  0.869 0.50        0.00         0.10
     info20       20.0           20  0.957 0.29        0.50         1.00
     info40       40.0           40  0.975 0.23        0.74         1.00
      se0.3       33.5           21  0.967 0.26        0.93         1.00
 se0.3_top5       33.6           21  0.965 0.27        0.93         0.86
 never_used
          0
        205
        144
        121
        114

Run on 25 September 2026 for 600 respondents, in 1.3 minutes.

Twenty items by information correlate 0.957 with the full bank (RMSE 0.29), against 0.869 (0.50) at random; forty reach 0.975. Under a target SE of 0.3, 93% reach it, in a mean of 33.5 items but a median of 21. The last two columns show the cost: twenty items by information use only 66 of the 271, and one goes to every student (86% under randomesque selection).

Code 
if (!is.null(cat600)) {
  b <- cat600$by_band
  print(data.frame(full_bank_theta = b$band, students = b$respondents, mean_items = round(b$mean_items, 1),
                   share_at_cap = round(b$share_at_cap, 2), mean_post_sd = round(b$mean_se, 2)), row.names = FALSE)
}
 full_bank_theta students mean_items share_at_cap mean_post_sd
       (-Inf,-1]       90       12.6         0.00         0.29
          (-1,0]      211       16.0         0.00         0.30
           (0,1]      180       33.7         0.00         0.30
           (1,2]       99       75.8         0.22         0.30
        (2, Inf]       20      100.0         1.00         0.37

Students below \(\theta = -1\) need 12.6 items on average; between 1 and 2 they need 75.8, with 22% at the cap, and all 20 above 2 stop at 100 items with a posterior SD of 0.37. Choi et al. (2025) saw the same in their live CAT, where students at the extremes reached its 50-item maximum, and they call for more hard items. The post-hoc CAT finds this before anyone is tested. (Their bank pooled six sittings whose IRW tables share no items; how those were put on one scale is a question for Linking and equating.)

What does the choice of stopping rule rest on? Two assumptions: whether scores must be compared at equal precision, and whether everyone must get the same testing time. A target standard error with a cap takes the first and gives up the second; a fixed length does the reverse. A post-hoc CAT on real responses, as here, shows what each rule costs for a given bank, and where the bank runs out before any respondent does. For this bank, the students above \(\theta = 2\) reach the 100-item cap with a posterior SD of 0.37, which says that new hard items would buy more than a higher cap.

PROMIS depression: a bank built for symptoms

For contrast, promis1wave1_depression: the 56 candidate depression items from the first wave of PROMIS testing, given to US adults (Cella et al., 2010; data at https://doi.org/10.7910/DVN/0NGAKG). Its job is to show a bank whose information sits where its purpose puts it. The items ask how often, in the past seven days, the respondent had a depressive feeling or thought, from never to always; PROMIS kept 28 in its bank (Pilkonis et al., 2011). HealthMeasures, which distributes PROMIS, doesn’t permit republishing the items, so we don’t quote them. Higher means more frequent symptoms; we recode the categories 0–4.

Recall from Models for polytomous responses: the graded response model gives each category boundary a 2PL with the item’s slope, so a five-category item is informative near each of its four boundaries. PROMIS calibrated its banks with it.

Code 
# PROMIS wave 1 depression: 56 candidate items, each on five frequency categories, 1
# (never) to 5 (always), so higher means more symptoms; none needs reversing. Recoded 0-4.
dep_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v60_0.promis1wave1_depression/rows?format=csv"
dd <- read.csv(dep_url)
dep <- tapply(dd$resp - 1, list(dd$id, dd$item), function(x) x[1])
dep <- dep[, order(colnames(dep))]
n_answered <- rowSums(!is.na(dep))
c(respondents = nrow(dep), items = ncol(dep), answered_7 = sum(n_answered == 7),
  answered_all_56 = sum(n_answered == 56))
    respondents           items      answered_7 answered_all_56 
          14909              56           13975             781
Code 
round(mean(dep == 0, na.rm = TRUE), 2)   # share of responses that are "never"
[1] 0.52

Most respondents answered 7 items; 781 answered all 56. We keep 300 of those for a post-hoc CAT and calibrate the graded response model on everyone else; the engine works unchanged.

Code 
# 300 of the respondents who answered all 56 items take the CAT; the graded response
# model (PROMIS's own choice) is calibrated on everyone else, with the missing blocks
# left missing.
set.seed(56)
dep_cat <- sample(which(n_answered == 56), 300)
m_dep <- mirt(as.data.frame(dep[-dep_cat, ]), 1, itemtype = "graded", verbose = FALSE)
bank_dep <- make_bank(m_dep, colnames(dep))
X_dep <- dep[dep_cat, ]
I_dep <- rowSums(sapply(colnames(dep), function(i) iteminfo(extract.item(m_dep, i), th)))
c(peak_at = th[which.max(I_dep)], peak_info = round(max(I_dep), 1))
  peak_at peak_info 
     1.53    129.80
Code 
round(sapply(c(-2, -1, -0.5, 0, 1, 2, 3), function(t) 1 / sqrt(I_dep[abs(th - t) < 1e-9])), 2)
[1] 0.49 0.19 0.13 0.10 0.09 0.09 0.13
Code 
b_dep <- coef(m_dep, IRTpars = TRUE, simplify = TRUE)$items
round(quantile(b_dep[, "b1"], c(0, 0.5, 1)), 2)   # the lowest boundary, never vs. rarely
   0%   50%  100% 
-1.20 -0.04  2.70

Information peaks at \(\theta = 1.53\) (CSEM 0.09), and the CSEM stays below 0.2 from \(-1\) to 3; at \(-2\), all 56 items together give 0.49. The median item’s lowest boundary, between never and rarely, is at −0.04: most items start to be informative only where symptoms start. Pilkonis et al. (2011) describe the calibrated banks as more informative than older measures from about \(-1\) to \(+3\) SDs.

Code 
# Post-hoc CAT to a posterior SD of 0.3 (no cap beyond the 56 items), against each
# respondent's full-bank EAP.
full_dep <- t(apply(X_dep, 1, function(x) unlist(run_cat(x, bank_dep, max_items = 56)[1:3])))
set.seed(57)
cat_dep <- t(apply(X_dep, 1, function(x) unlist(run_cat(x, bank_dep, max_items = 56, se_stop = 0.3)[1:3])))
c(median_items = median(cat_dep[, "n"]), mean_items = round(mean(cat_dep[, "n"]), 1),
  r_full = round(cor(cat_dep[, "theta"], full_dep[, "theta"]), 3),
  reached_0.3 = round(mean(cat_dep[, "se"] <= 0.3), 2))
median_items   mean_items       r_full  reached_0.3 
       5.000       10.000        0.954        0.910
Code 
band <- cut(full_dep[, "theta"], c(-Inf, -0.5, 0.5, 1.5, Inf))
data.frame(respondents = as.vector(table(band)),
           mean_items = round(tapply(cat_dep[, "n"], band, mean), 1),
           mean_post_sd = round(tapply(cat_dep[, "se"], band, mean), 2))
respondents mean_items mean_post_sd
(-Inf,-0.5] 120 19.6 0.31
(-0.5,0.5] 111 3.7 0.28
(0.5,1.5] 49 2.6 0.28
(1.5, Inf] 20 5.8 0.29
Code 
c(at_floor = sum(rowSums(X_dep) == 0), floor_eap = round(unique(full_dep[rowSums(X_dep) == 0, "theta"]), 2))
 at_floor floor_eap 
     7.00     -2.22

The CAT needs a median of 5 items and a mean of 10.0, and correlates 0.95 with the full bank (Choi et al., 2010, found CAT and full-bank scores highly correlated on the 28-item bank). By band the lengths run the other way from the CMSCE’s: 2.6 to 3.7 items between \(-0.5\) and 1.5, and 19.6 below \(-0.5\), where the average posterior SD at the stop is still 0.31. The seven respondents who chose “never” on all 56 items share an EAP of −2.2. None of this is a fault in the respondents or the items: a depression bank is built to tell mild from moderate from severe, which it does in three or four items, and telling “none” from “very few” isn’t its job. The examination is thin at the top, the depression bank at the bottom, and in both a post-hoc CAT finds the thin end.

Problems

  1. Which item next? (a) Under the Rasch model, show that the most informative item at \(\hat\theta\) has \(b\) closest to \(\hat\theta\). (b) Under the 2PL, which of \(a = 2, b = 1\) and \(a = 1, b = 0\) is more informative at \(\hat\theta = 0\), and where does the answer change? Why does maximum information give steep items early?
  2. A stricter bank. Apply Choi et al.’s (2025) screen (slopes above 0.6, \(|b| < 5\)) to the CMSCE calibration and rerun the target-SE CAT on the 50 live respondents. How much bank is left, and what happens at the top?
  3. You work at the state agency. You have last year’s summative test data for one grade: every student’s scored responses to a fixed form, and nothing else about the students. Build an item bank from it: which fields does each item’s record need, how would you calibrate, and how would next year’s new items join? Then say what you can’t do with these data, and what you would ask for. (There are no data for this problem.)
  4. The low end. A general-population survey expects most respondents to report few depressive symptoms. Would you use the PROMIS CAT, a fixed short form, or new low-end items first, and how would you calibrate new items onto the bank’s scale?
  5. Pass or fail. Put a pass mark at the 10th percentile of the 600 CMSCE students’ full-bank \(\theta\) and stop with an SPRT (\(\delta = 0.2\), error rates of 5%), choosing items by information at the cut. How many items does a decision take, by band, against the target-SE rule, and how many students are classified differently from their full-bank \(\theta\)?
  6. Challenge (open): post-hoc and live. A post-hoc CAT reuses responses given in a fixed form and order. How might responses in a live CAT differ (item position, fatigue, a test that feels hard to strong respondents, stakes)? Design a study that would show whether post-hoc results carry over. I don’t think one study settles it for every bank.

Stuck, or curious about something this lesson doesn’t cover? Ask Claude about this lesson opens a new conversation that already knows what the lesson is about; add your question at the end. Claude shows a caution notice on any prompt that arrives through a link; that’s expected.

Going further

  • Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., & Mislevy, R. J. (2000). Computerized adaptive testing: A primer (2nd ed.). Erlbaum. https://doi.org/10.4324/9781410605931. The standard introduction, from item banks to scoring.
  • van der Linden, W. J., & Glas, C. A. W. (Eds.). (2010). Elements of adaptive testing. Springer. https://doi.org/10.1007/978-0-387-85461-8. Item selection, exposure control, stopping and item pools, chapter by chapter.
  • Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6(4), 473–492. https://doi.org/10.1177/014662168200600408. Adaptive tests designed for equal precision along the scale.
  • Weiss, D. J., & Kingsbury, G. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x. Stopping rules for measurement and for classification.
  • Owen, R. J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70(350). https://doi.org/10.2307/2285821. An early Bayesian procedure for choosing items.
  • Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431–444. https://doi.org/10.1177/014662168200600405. EAP scoring for adaptive tests.
  • van der Linden, W. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2), 201–216. https://doi.org/10.1007/BF02294775. Selection criteria built on the full posterior (the Go deeper callout).
  • Chang, H.-H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3), 213–229. https://doi.org/10.1177/014662169602000303. Why Fisher information at the estimate suits the start of a test least.
  • Kingsbury, G. G., & Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2(4), 359–375. https://doi.org/10.1207/s15324818ame0204_6. Content balancing and exposure in item selection.
  • Chang, H.-H., & Ying, Z. (1999). a-Stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23(3), 211–222. https://doi.org/10.1177/01466219922031338. Holding steep items back.
  • Chang, H.-H. (2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80(1), 1–20. https://doi.org/10.1007/s11336-014-9401-5. A survey of the theory, and of problems met in large-scale use.
  • Hetter, R. D., & Sympson, J. B. (1997). Item exposure control in CAT-ASVAB. In W. A. Sands, B. K. Waters, & J. R. McBride (Eds.), Computerized adaptive testing: From inquiry to operation (pp. 141–144). American Psychological Association. https://doi.org/10.1037/10244-014. The Sympson–Hetter method in an operational test.
  • van der Linden, W. J. (2003). Some alternatives to Sympson-Hetter item-exposure control in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 28(3), 249–265. https://doi.org/10.3102/10769986028003249. What the Sympson–Hetter method does, and alternatives to it.
  • Revuelta, J., & Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35(4), 311–327. https://doi.org/10.1111/j.1745-3984.1998.tb00541.x. Eight selection methods compared for exposure and precision.
  • Wald, A. (1945). Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics, 16(2), 117–186. https://doi.org/10.1214/aoms/1177731118. The sequential probability ratio test.
  • Spray, J. A., & Reckase, M. D. (1996). Comparison of SPRT and sequential Bayes procedures for classifying examinees into two categories using a computerized test. Journal of Educational and Behavioral Statistics, 21(4), 405–414. https://doi.org/10.3102/10769986021004405. The SPRT as a CAT stopping rule.
  • Eggen, T. J. H. M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement, 23(3), 249–261. https://doi.org/10.1177/01466219922031365. Choosing items for a classification CAT.
  • Luecht, R. M., & Nungester, R. J. (1998). Some practical examples of computer-adaptive sequential testing. Journal of Educational Measurement, 35(3), 229–249. https://doi.org/10.1111/j.1745-3984.1998.tb00537.x. Multistage testing with preassembled modules.
  • Mislevy, R. J., & Wu, P.-K. (1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing. ETS Research Report Series, 1996(2). https://doi.org/10.1002/j.2333-8504.1996.tb01708.x. Why the items a CAT didn’t give can be ignored.
  • Binet, A., & Simon, T. (1904). Méthodes nouvelles pour le diagnostic du niveau intellectuel des anormaux. L’Année psychologique, 11, 191–244. https://doi.org/10.3406/psy.1904.3675. Tasks graded by age, a paper ancestor of adaptive testing.
  • Ma, W. A., Richie-Halford, A., Burkhardt, A. K., Kanopka, K., Chou, C., Domingue, B. W., & Yeatman, J. D. (2025). ROAR-CAT: Rapid Online Assessment of Reading ability with Computerized Adaptive Testing. Behavior Research Methods, 57(1), 56. https://doi.org/10.3758/s13428-024-02578-y. Checking a bank’s parameters across groups, and a randomized test of CAT against random order.
  • Choi, J., Jung, S.-S., Choi, E. K., Kim, K. S., & Seo, D. G. (2025). Feasibility of applying computerized adaptive testing to the Clinical Medical Science Comprehensive Examination in Korea: A psychometric study. Journal of Educational Evaluation for Health Professions, 22, 29. https://doi.org/10.3352/jeehp.2025.22.29. The live CAT built from the CMSCE.
  • Pilkonis, P. A., Choi, S. W., Reise, S. P., Stover, A. M., Riley, W. T., Cella, D., & PROMIS Cooperative Group. (2011). Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS®): Depression, anxiety, and anger. Assessment, 18(3), 263–283. https://doi.org/10.1177/1073191111411667. How the depression bank was built and calibrated.
  • Choi, S. W., Reise, S. P., Pilkonis, P. A., Hays, R. D., & Cella, D. (2010). Efficiency of static and computer adaptive short forms compared to full-length measures of depressive symptoms. Quality of Life Research, 19(1), 125–136. https://doi.org/10.1007/s11136-009-9560-5. Post-hoc CATs and short forms on the PROMIS depression bank.
  • Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194. https://doi.org/10.1016/j.jclinepi.2010.04.011. The source of the depression data.
  • Magis, D., & Raîche, G. (2012). Random generation of response patterns under computerized adaptive testing with the R package catR. Journal of Statistical Software, 48(8). https://doi.org/10.18637/jss.v048.i08. A package for simulating CATs, with many selection and stopping rules.
  • Chalmers, R. P. (2016). Generating adaptive and non-adaptive test interfaces for multidimensional item response theory applications. Journal of Statistical Software, 71(5). https://doi.org/10.18637/jss.v071.i05. mirtCAT, for running CATs, live or post hoc, from mirt models.
  • Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6). https://doi.org/10.18637/jss.v048.i06. The calibrations, and the item probabilities and information the engine uses.
  • Lord, F. M. (1980). Applications of item response theory to practical testing problems. Erlbaum. https://doi.org/10.4324/9780203056615. Chapter 10 is on tailored testing.
  • van der Linden, W. J. (2017). Adaptive testing. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 3. Applications (pp. 197–228). Chapman and Hall/CRC. https://doi.org/10.1201/9781315117430. A handbook chapter on adaptive testing.
  • Gershon, R. C., Hays, R. D., & Kallen, M. A. (2017). Health measurement. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 3. Applications (pp. 349–364). Chapman and Hall/CRC. https://doi.org/10.1201/9781315117430. A handbook chapter on item response theory in health measurement.

Data sources

Data from the Item Response Warehouse (each table name links to its IRW page); references from IRW biblio.

  • choi_2026_cmsce_2020_1: Choi, J., Jung, S.-S., Choi, E. K., Kim, K. S., & Seo, D. G. (2026). Feasibility of applying computerized adaptive testing to the Clinical Medical Science Comprehensive Examination in Korea: a psychometric study [Data set]. Harvard Dataverse. https://doi.org/10.7910/DVN/TQZQ6L. Licence: CC0 1.0.
  • promis1wave1_depression: Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., … & Hays, R. (2010). Initial adult health item banks and first wave testing of the patient-reported outcomes measurement information system (PROMIS™) network: 2005–2008. Journal of clinical epidemiology, 63(11), 1179. Cella, David, 2015, “PROMIS 1 Wave 1”, https://doi.org/10.7910/DVN/0NGAKG, Harvard Dataverse, V3, UNF:6:sXYqKDzGWfX7QuTXNGpjGQ== [fileUNF]. Licence: CC0 1.0.

For instructors

  • Source: EDUC 252 class 6 (slides c6, 22–31: item banks, the adaptive loop, ROAR-CAT) and class 10 (slides c10, 44–45 and 49: item banks in practice, and the state-agency exercise). There was no 252 code or problem set for this lesson; the CAT engine, the widgets, the simulation and the post-hoc CATs are new.
  • Timing: the core ideas and widgets take about 50 minutes; the two real-data analyses work as a 30-minute lab. The page’s code takes about two and a half minutes in local R, most of it the two calibrations; compute.R takes under two minutes.
  • Precomputed results: the 600-respondent CAT under five rules is in deepdives/item-banks-cat/results.rds, written by compute.R (digest E12); the page runs the first 50 respondents itself. compute.R runs this page’s own code for the data, the calibration and the engine, so the two can’t drift apart.
  • Changed from 252: the slides walk through a CAT on a figure; here the reader steps through one in a widget and then runs one on real responses. The state-agency exercise (c10, slide 45) pointed to anonymized state data that isn’t in the IRW, so it is problem 3, a design problem without data (digest F21). A paper basal-and-ceiling widget in the outline was dropped for length; basal and ceiling rules get a sentence.
  • Held back: writing and pre-calibrating new items, and predicting their difficulty from the text (AI and psychometrics); multidimensional CAT (Dimensionality and multidimensional IRT); response times in adaptive tests (Response time and the speed-accuracy tradeoff). Item drift over time, multistage testing and content-balancing algorithms are named but not taught.
  • Item text: PROMIS items are summarized, not quoted (HealthMeasures’ terms of use bar redistribution). The CMSCE deposit holds scored responses only, with no item text.
  • Solutions: held back in solutions/item-banks-cat.qmd (not published).