Missing responses: omitted, not reached, and ignorable

Module: Item response theory
Tranche: extension (beyond a first course)
Before this: Estimating abilities: MLE and EAP
Builds toward: none
Picks up threads: EAP estimates shrink toward the mean (from Estimating abilities: MLE and EAP); No maximum-likelihood estimate for all-right or all-wrong patterns (from Estimating abilities: MLE and EAP); Why a response is missing (skipped, not shown, stopped early) (from Item response data and the IRW)
IRW tables: mthimkhulu_2023_pirls_reading, pirlsmissing_sirt, borges_brazil_residency_2024_cbt
From EDUC 252: n/a
Status: draft
Authorship: Written largely by Claude (Anthropic), from Ben Domingue’s EDUC 252 materials and under Ben’s direction. Ben reviews each lesson before it is marked done.
Comments: open an issue on GitHub about this lesson.

What this is for

A fourth grader answers the first eight questions on a reading passage, leaves the ninth blank, answers the tenth and eleventh, and writes nothing after that. What should the blank on item 9 count for, and the four at the end? Most analyses in this course have left gaps missing, a choice that rests on why a response is missing. This lesson separates responses that are omitted (seen and skipped) from those not reached (never got to), recovers the two from the order of the items in an IRW table that doesn’t record them, and asks, in PIRLS 2023 reading data, which kind carries information about ability. It picks up from Estimating abilities, because the EAP’s pull toward the mean is part of what a missing response does to a score.

Goals

By the end of this lesson you should be able to:

  • classify a kind of missing response as MCAR, MAR or MNAR, and say what the long table can’t tell you;
  • explain why marginal maximum likelihood can ignore gaps left by a booklet design or a stopping rule, and when it can’t;
  • show what scoring an omit as wrong, or leaving it missing, does to one respondent’s estimate;
  • recover omitted and not-reached responses from item order, and check the rule against a source that records them;
  • fit the same data under several treatments of the gaps, including a response-propensity model, and report how far the estimates move.

Core ideas

Why a response is missing is part of the data

Recall from Item response data and the IRW: a missing response is an absent row in a long table, and it becomes an empty cell when the table is reshaped. Skipped at random, missing by booklet design and stopped early leave different patterns of gaps, and the table alone doesn’t say which produced a given gap.

Write \(m_{ij} = 1\) when respondent \(j\)’s response to item \(i\) is missing, so that every cell carries a pair \((x_{ij}, m_{ij})\), with \(x_{ij}\) unseen when \(m_{ij} = 1\). Rubin (1976) sorted the ways the \(m_{ij}\) can arise into three kinds (Little & Rubin, 2019; Schafer & Graham, 2002):

  • Missing completely at random (MCAR): missingness has nothing to do with the respondent or the item, as when a page is lost in the post.
  • Missing at random (MAR): it depends only on what we observe. A booklet design, an adaptive test (the next item depends on the answers so far) and a stopping rule on the first half of a test are MAR.
  • Missing not at random (MNAR): it depends on something unobserved, such as the response itself (“skip it if I’d get it wrong”) or \(\theta\) (weaker readers skip more).

The labels describe the process that produced the gaps, not the gaps (Seaman, Galati, Jackson & Carlin, 2013, state them with care). The design and the source’s missing codes say which kind we have; the long table rarely does.

Try it: three mechanisms

2,000 simulated respondents answer twelve Rasch items (\(b\) from −2 to 2). Choose how responses go missing; the grid shows 30 of them, sorted by true \(\theta\).

Under MCAR the observed responses are a fair sample: 0.50 correct, as in the complete data. Under the other three the observed proportion correct rises (to 0.61, 0.55 and 0.63 at the default share), because the missing responses would more likely have been wrong. On this summary the stopping rule looks no better than the two MNAR mechanisms; what separates them is what a model can do about it.

Ignorable: when the likelihood can leave the gaps alone

Why does the mechanism matter for a fitted model? Rubin (1976) showed that if the gaps are MAR, and the parameters of the missingness process are distinct from the model’s, the likelihood of the observed responses alone gives the right answer: the process is ignorable (Mislevy & Wu, 1996, and Mislevy, 2017, work this out for item response models). The marginal likelihood that mirt maximizes (Bock & Aitkin, 1981; Chalmers, 2012) is that likelihood: an empty cell contributes nothing. So booklet designs and adaptive tests are ignorable, although an adaptive test’s gaps depend on ability through the answers already given. The Go deeper callout writes out the argument.

Deleting incomplete respondents (listwise deletion) is also unbiased under MCAR, at the cost of the data it discards; in a booklet design it discards everyone (in the TIMSS 2007 data checked below, no student saw every item). Finch (2008) compares treatments of missing responses for item parameters.

If you’ve done Confirmatory factor analysis: lavaan’s missing = "ml" (full-information ML) is the same idea for factor models (Enders & Bandalos, 2001). If you’ve done Estimating item parameters: EM treats \(\theta\) itself as missing data.

Assumptions: responses \(x\) follow an item response model with item parameters \(\beta\) and a population distribution \(g(\theta)\); the indicators \(m\) follow a model with parameters \(\phi\); write \(x = (x_{\text{obs}}, x_{\text{mis}})\). (Here \(\beta\) collects every item parameter, not regression coefficients.)

The factorization. The observed data are \((x_{\text{obs}}, m)\), and their probability averages over what we didn’t see: \[p(x_{\text{obs}}, m \mid \beta, \phi) = \int\!\!\int p(x_{\text{obs}}, x_{\text{mis}} \mid \theta, \beta)\, p(m \mid x_{\text{obs}}, x_{\text{mis}}, \theta, \phi)\, g(\theta)\, dx_{\text{mis}}\, d\theta.\] Under MAR, \(p(m \mid x_{\text{obs}}, x_{\text{mis}}, \theta, \phi) = p(m \mid x_{\text{obs}}, \phi)\): the gaps depend only on what we observe. That factor then leaves both integrals, and \[p(x_{\text{obs}}, m \mid \beta, \phi) = p(m \mid x_{\text{obs}}, \phi) \int p(x_{\text{obs}} \mid \theta, \beta)\, g(\theta)\, d\theta.\] If \(\phi\) and \(\beta\) are distinct (no restriction ties them), the first factor is a constant as far as \(\beta\) is concerned, and maximizing the second, the marginal likelihood of the observed responses, is all we need.

Where it breaks. If the gaps depend on \(\theta\), as when lower-ability respondents skip more, then \(p(m \mid \cdot)\) contains \(\theta\) and can’t leave the integral over it. For the marginal likelihood this is MNAR, although no single response is hidden from the mechanism. It becomes ignorable again if we add a second latent variable \(\xi\) for the propensity to respond, correlated with \(\theta\): given \((\theta, \xi)\), the gaps depend on nothing unobserved. That is the premise of the propensity models in this lesson (Holman & Glas, 2005). Gaps that depend on the unseen response itself (“skip it if wrong”) remain MNAR even then.

Why it is load-bearing. Concurrent calibration across booklets (Linking and equating), adaptive testing (Item banks and adaptive testing) and a skip node in a tree (IRTrees) all rest on this factorization.

Omitted: wrong, missing, or something in between?

A student who reads an item and leaves it blank has omitted it. The two simple treatments are each an assumption: scoring it wrong assumes the student would have got it wrong, and leaving it missing assumes the omit says nothing about \(\theta\). Lord (1974) argued that the first is not justified when estimating abilities or item parameters, and gave a method that uses what omits reveal. Treatments are compared in simulation by De Ayala, Plake and Impara (2001) and reviewed for large-scale assessments by Robitzsch (2021).

Recall from Estimating abilities: the EAP is pulled toward the prior mean, most where the likelihood is flattest, which includes short tests; and a respondent who gets every item right has no maximum-likelihood estimate.

Leaving an omit missing shortens the respondent’s test, so both apply. A respondent who answers only the items they know can have a perfect pattern on those items, and no MLE; their EAP is pulled toward the population mean, which moves a below-average respondent up.

Try it: one respondent, omits scored two ways

Ten Rasch items (\(b\) from −2 to 2, easiest first), normal(0, 1) prior. Tick the items right and the items omitted (an omit overrides a “correct” tick); the rest are wrong.

With three right, two wrong and five omitted, the EAP is −0.73 when the omits are wrong and −0.31 when they are missing. Tick items 4 and 5 as omitted too: left missing, the three answered items are all right, the MLE is \(+\infty\), and the EAP rises to 0.45, though the respondent answered only the three easiest items. The posterior SD grows as items drop out.

Modelling the propensity to respond

If omitting goes with ability, can a model use it? The latent-ignorability models add a second latent variable, the response propensity \(\xi_j\), which drives the indicators \(m_{ij}\) and correlates with \(\theta_j\) (Holman & Glas, 2005; Rose, von Davier & Xu, 2010). Given both, the gaps are ignorable, and the correlation carries what omitting says about \(\theta\). The model of Mislevy and Wu (1996) also lets an omit depend on the response the student would have given; scoring omits wrong and latent ignorability are special cases of it (Rose, von Davier & Nagengast, 2017). Each rests on an assumption about unseen responses that the data can’t check directly. On the German National Educational Panel Study’s competence tests, the model-based treatments and leaving the gaps missing gave similar results, and scoring omits wrong different ones (Pohl, Gräfe & Rose, 2014; Köhler, Pohl & Carstensen, 2015).

If you’ve done IRTrees: a first node “answered?” is a propensity model (Debeer, Janssen & De Boeck, 2017). There, simulated heavy skippers came out 0.24 too low with skips wrong, 0.41 too high with skips missing, and 0.02 off with the skip node.

Try it: the propensity dial

400 simulated respondents answer 16 Rasch items and omit about a fifth; their propensity to respond correlates with \(\theta\) at the value you choose. Each is scored by EAP three ways, with the true parameters known. The bars show the average error (\(\hat\theta - \theta\)) for the quarter who omit most.

At a correlation of 0, leaving omits missing is right on average (−0.02), the propensity model agrees, and scoring them wrong puts heavy omitters about a logit too low. As the correlation rises, the “missing” estimates drift up (0.24 at 0.5, 0.42 at 0.8), while the propensity model stays within 0.04 of zero. Scoring wrong stays furthest off, since these respondents omit without regard to whether they’d be right.

Not reached: time, not knowledge

A not-reached item sits at the end of a timed part the respondent never finished, and it says something about speed, which may or may not go with ability. The IEA, which runs TIMSS and PIRLS, defines it by position: within a part of the booklet, an unanswered item is not reached if the item before it is also unanswered and no later item in the part is answered (Foy, Galia & Li, 2008, §11.3.3). So the first blank of a trailing run counts as omitted, and a run of one is an omit. TIMSS 2007 treated not-reached items as not administered when it estimated item parameters, and as wrong when it scored students (the same section).

Why treat them differently for items? Scored wrong, an item late in a timed test looks harder than it is. In TIMSS 2003 the same item looked harder when it came late in a booklet than when it came early (the same report, §11.3.1), and Oshima (1994) showed the effect in simulation. Speeded respondents can also be modelled as a latent class (Bolt, Cohen & Wollack, 2002) or with a propensity for speed (Glas & Pimentel, 2008); van der Linden (2018) treats speededness in full.

The IEA rule needs only the order in which items were presented and where each timed part ends. The IRW keeps the gaps, so where the order is known the rule works on a table that records no codes at all.

Try it: read the order

The PIRLS 2023 learners from the main table below, in item order: 150 of the 1,894 are plotted, sorted by their last answer, and the counts use everyone. Choose a rule for the gaps.

Learners lower down the plot stop earlier. Calling every gap an omit agrees with the source on 43.5% of the coded cells; calling the whole trailing run not reached, 89.5% (it misses wherever a learner’s first trailing blank was a skip); the IEA rule, 100%.

Simulate

What does each treatment do when we know the truth? The code below simulates Rasch responses (1,000 respondents, 20 items), deletes some by four mechanisms like those in the first widget, and fits the Rasch model with the gaps missing and with them scored wrong. It reports the average and worst error in \(b\), and the average error in the EAP for the tenth of respondents with the most gaps. The first run takes 20–30 seconds while mirt loads.

With the seed as written, leaving the gaps missing recovers \(b\) within 0.07 on average under MCAR, the stopping rule and the \(\theta\)-driven skips, and misses by −0.57 when respondents skip what they’d get wrong. Scoring the gaps wrong is right only in that last case, and pushes \(b\) up by half a logit under MCAR. For the heavy skippers under \(\theta\)-driven skipping, the EAPs come out 0.43 too high with the gaps missing and 0.46 too low with them wrong, though \(b\) is nearly right. Try the skip correlation at −0.3 or 0 (the -0.6 in the line that makes xi), or n <- 5000 locally.

Download this code to run it in your own R session.

With real data

Our main example is mthimkhulu_2023_pirls_reading: 1,894 South African fourth graders’ responses to the fifteen questions on “Octopuses”, a released informational passage from PIRLS 2023, shared by the University of Pretoria (Mthimkhulu, Roux & Mihai, 2023; CC BY 4.0). Its job is to be a fixed-order test on which a quarter of the responses are missing. Constructed-response items were scored by PIRLS raters (0–1, 0–2 or 0–3) and the four multiple-choice items against the key, so higher is better throughout. The IRW script drops the source’s missing codes, so a missing response is an absent row. Download the code to run every chunk yourself; every choice about missing data is marked in its comments.

Code 
library(mirt)
p23_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse_4:v9_0.mthimkhulu_2023_pirls_reading/rows?format=csv"
p23 <- read.csv(p23_url)
# One released passage, "Octopuses": items RP51Z01 to RP51Z15, numbered in the order
# they are printed. Constructed-response items are scored 0-1, 0-2 or 0-3 by PIRLS
# raters; the four multiple-choice items are scored 1 = the keyed option.
items <- sprintf("RP51Z%02d", 1:15)
ids <- sort(unique(p23$id))
Y <- matrix(NA, length(ids), 15, dimnames = list(ids, items))
Y[cbind(match(p23$id, ids), match(p23$item, items))] <- p23$resp
# Missing-data choice: a learner-item pair with no row in the table is a missing
# response. The IRW processing script drops rows whose source code is 6 (not
# reached) or 9 (omitted or invalid), so the table doesn't say which is which.
# Learners who answered nothing on this passage have no rows, so they aren't here.
c(learners = nrow(Y), items = ncol(Y), share_missing = round(mean(is.na(Y)), 3),
  no_gap = round(mean(rowSums(is.na(Y)) == 0), 2))
     learners         items share_missing        no_gap 
     1894.000        15.000         0.252         0.360
Code 
table(booklet = tapply(p23$cov_booklet, p23$id, `[`, 1))   # two booklets carry the passage
booklet
 12  17 
950 944

A quarter of the cells are empty, and 36% of learners have no gap. Is a gap an omit or a sign of running out of time? We treat the passage as one timed part and apply the IEA rule in item order.

Code 
# The IEA's rule, applied within one timed part: an unanswered item is not reached if
# the item before it is also unanswered and nothing after it is answered. So the
# first item of a trailing run of gaps counts as omitted, and a run of one is an omit.
classify <- function(miss) {           # miss: TRUE/FALSE per item, in the order given
  last <- max(c(0, which(!miss)))      # position of the last answered item
  nr <- seq_along(miss) > last + 1     # after the last answer, and after the first gap
  ifelse(!miss, "answered", ifelse(nr, "not reached", "omitted"))
}
# Missing-data choice: the whole passage is treated as one timed part, in item order.
code <- t(apply(is.na(Y), 1, classify))
dimnames(code) <- dimnames(Y)
table(code)
code
   answered not reached     omitted 
      21254        4043        3113
Code 
round(rbind(not_reached = colMeans(code == "not reached"),
            omitted = colMeans(code == "omitted")), 2)
            RP51Z01 RP51Z02 RP51Z03 RP51Z04 RP51Z05 RP51Z06 RP51Z07 RP51Z08
not_reached    0.00    0.00    0.01    0.03    0.05    0.08    0.10    0.12
omitted        0.07    0.13    0.19    0.17    0.11    0.10    0.09    0.17
            RP51Z09 RP51Z10 RP51Z11 RP51Z12 RP51Z13 RP51Z14 RP51Z15
not_reached    0.15    0.17    0.23    0.26    0.28    0.30    0.35
omitted        0.06    0.14    0.08    0.09    0.13    0.08    0.05
Code 
nr <- rowSums(code == "not reached"); om <- rowSums(code == "omitted")
c(any_not_reached = round(mean(nr > 0), 2), five_or_more = sum(nr >= 5))
any_not_reached    five_or_more 
           0.35          427.00

The rule finds 4,043 not-reached cells and 3,113 omits. Not reached climbs along the passage from 0 on the first two items to 35% on the last; omits stay between 5% and 19%. A third of learners (35%) have at least one not-reached item, and 427 have five or more.

The source file keeps the codes, so we can check the rule. It also carries each learner’s first plausible value, PIRLS’s reading score from the whole booklet: an ability measure from outside these fifteen items.

Code 
# A check against the source file (CC BY 4.0), which keeps codes 6 and 9. It also
# carries the learners' first plausible value, PIRLS's own reading score from the
# whole booklet (two passages), which we use as an ability measure from outside the
# 15 items. Needs the haven package.
sav <- tempfile(fileext = ".sav")
download.file("https://ndownloader.figshare.com/files/43586289", sav, mode = "wb", quiet = TRUE)
src <- haven::read_sav(sav, user_na = TRUE)   # user_na = TRUE keeps codes 6 and 9
src <- src[match(ids, as.numeric(src$IDSTUD)), ]
raw <- sapply(items, function(v) as.numeric(unclass(src[[v]])))
source_code <- ifelse(raw %in% 6, "6 not reached", ifelse(raw %in% 9, "9 omitted",
                 ifelse(is.na(raw), "blank in source", "answered")))
table(source = source_code, rule = code)
                 rule
source            answered not reached omitted
  6 not reached          0        4032       0
  9 omitted              0           0    3107
  answered           21254           0       0
  blank in source        0          11       6
Code 
pv1 <- as.numeric(src$ASRREA01)

The rule reproduces the source’s codes on every cell: 4,032 of 4,032 not reached and 3,107 of 3,107 omitted. The 17 cells blank in the source, with neither code, come out as 11 not reached and 6 omitted. With the item order, the IRW table holds everything the codes would have told us.

Code 
# Does omitting or not reaching go with reading ability?
round(c(omitted = cor(om, pv1), not_reached = cor(nr, pv1)), 2)
    omitted not_reached 
      -0.16       -0.02
Code 
fifth <- cut(pv1, quantile(pv1, 0:5 / 5), include.lowest = TRUE, labels = 1:5)
round(rbind(omitted = tapply(om / 15, fifth, mean),
            not_reached = tapply(nr / 15, fifth, mean)), 3)   # share of the 15 items
                1     2     3     4     5
omitted     0.131 0.124 0.117 0.105 0.071
not_reached 0.113 0.147 0.167 0.177 0.107

Omits track ability and not-reached items hardly do: −0.16 against −0.02. The lowest fifth omits 13.1% of the items and the highest 7.1%, while not reached rises from 11.3% to 17.7% across the first four fifths and falls back to 10.7% in the top one. Skipping leans toward the bottom of the ability range; running out of time sits in its middle.

What do the treatments do? We fit the partial credit model (Masters, 1982) with every gap wrong, every gap missing, and a split by the rule (rule in the code): omits 0, and not-reached items missing, as TIMSS calibrates them.

Code 
# Three ways to score the same gaps, each fitted with the partial credit (Rasch)
# model. mirt's "Rasch" item type gives the partial credit model for items with more
# than two categories.
Y_wrong <- Y;   Y_wrong[is.na(Y)] <- 0              # every gap scored 0
Y_missing <- Y                                      # every gap left missing
Y_rule <- Y;    Y_rule[code == "omitted"] <- 0      # omits 0, not reached missing
fits <- lapply(list(wrong = Y_wrong, missing = Y_missing, rule = Y_rule),
               function(Z) mirt(as.data.frame(Z), 1, itemtype = "Rasch", verbose = FALSE))
# An item's location: the mean of its step difficulties (b for a 0/1 item).
loc <- function(f) {
  cf <- coef(f, IRTpars = TRUE, simplify = TRUE)$items
  rowMeans(cf[, grep("^b", colnames(cf)), drop = FALSE], na.rm = TRUE)
}
b <- sapply(fits, loc)
round(c(last5 = mean(b[11:15, "wrong"] - b[11:15, "rule"]),
        first5 = mean(b[1:5, "wrong"] - b[1:5, "rule"])), 2)
 last5 first5 
  0.40   0.06
Code 
# EAP abilities under each scoring, standardized within each scoring.
th <- scale(sapply(fits, function(f) fscores(f, method = "EAP")[, 1]))
rbind(nr5 = colMeans(th[nr >= 5, ]), om4 = colMeans(th[om >= 4, ])) |> round(2)
    wrong missing  rule
nr5 -0.51   -0.10 -0.20
om4 -0.49   -0.31 -0.56
Code 
c(n_nr5 = sum(nr >= 5), n_om4 = sum(om >= 4))
n_nr5 n_om4 
  427   322

With every gap wrong, the last five items sit 0.40 logits higher on average than under the split, and the first five only 0.06 higher: the late items absorb the time limit. The 427 learners with five or more not-reached items average −0.51 SD with every gap wrong, −0.10 with every gap missing and −0.20 under the split. For the 322 learners with four or more omits, the last two change places: −0.31 with gaps missing, −0.56 under the split.

A propensity model gives the omits a second latent variable; not-reached items stay missing on both the item and its indicator.

Code 
# A response-propensity model: a second latent variable (xi) drives whether a reached
# item is answered, and correlates with theta. Missing-data choice: an omit is a 0 on
# its indicator; a not-reached item is missing on both the item and its indicator.
answered <- ifelse(code == "not reached", NA, ifelse(code == "omitted", 0, 1))
colnames(answered) <- paste0("ans_", 1:15)
two <- mirt.model("theta = 1-15\n xi = 16-30\n COV = theta*xi")
fit_prop <- mirt(data.frame(Y_missing, answered), two, itemtype = "Rasch", verbose = FALSE)
S <- coef(fit_prop, simplify = TRUE)$cov
round(c(cor_theta_xi = S[1, 2] / sqrt(S[1, 1] * S[2, 2])), 2)
cor_theta_xi 
         0.3
Code 
th_prop <- scale(fscores(fit_prop, method = "EAP")[, 1])
round(c(om4_propensity = mean(th_prop[om >= 4])), 2)
om4_propensity 
         -0.46

The propensity to answer a reached item correlates 0.30 with reading ability, and the heavy omitters’ mean estimate is −0.46 SD: between leaving their omits missing (−0.31) and scoring them 0 (−0.56), nearer the second.

A contrast: when the codes were merged

Our contrast table is pirlsmissing_sirt, one PIRLS 2011 booklet (the PIRLS Reader) taken by fourth graders in Austria, Germany, France and the Netherlands, released with the sirt package (Robitzsch, 2022). Its job is to show the rule where the source had already merged omitted and not reached into one code. Responses are 1 when correct; each passage is taken to be its own timed part, still to be checked against the PIRLS 2011 design.

Code 
# Contrast: PIRLS 2011, one booklet (the PIRLS Reader), four countries. The source
# had already merged omitted and not reached into one code, which the IRW keeps as
# rows with an empty resp.
p11 <- read.csv("https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v64_0.pirlsmissing_sirt/rows?format=csv")
# Missing-data choice: R31G08CZ and R31G13CZ combine the lettered parts of their
# question, so we drop them and keep the parts.
p11 <- p11[!grepl("CZ$", p11$item), ]
items11 <- sort(unique(p11$item)); ids11 <- sort(unique(p11$id))
X <- matrix(NA, length(ids11), length(items11), dimnames = list(ids11, items11))
X[cbind(match(p11$id, ids11), match(p11$item, items11))] <- p11$resp
country <- p11$country[match(ids11, p11$id)]
# Missing-data choice: each passage (G, then P) is its own timed part, items in
# number order.
code11 <- X; code11[] <- NA
for (part in list(grep("^R31G", items11), grep("^R31P", items11)))
  code11[, part] <- t(apply(is.na(X[, part]), 1, classify))
round(prop.table(table(code11)), 3)
code11
   answered not reached     omitted 
      0.915       0.013       0.072
Code 
pc <- rowSums(X, na.rm = TRUE) / rowSums(!is.na(X))   # proportion correct on answered items
round(c(omitted = cor(rowSums(code11 == "omitted"), pc),
        not_reached = cor(rowSums(code11 == "not reached"), pc)), 2)
    omitted not_reached 
      -0.26       -0.14
Code 
# The same Netherlands-France comparison under three scorings (Rasch, pooled, no
# weights: a check on the scoring, not an estimate of either country's performance).
X_wrong <- X; X_wrong[is.na(X)] <- 0
X_rule <- X;  X_rule[code11 == "omitted"] <- 0
gap <- sapply(list(wrong = X_wrong, missing = X, rule = X_rule), function(Z) {
  s <- fscores(mirt(as.data.frame(Z), 1, itemtype = "Rasch", verbose = FALSE))[, 1]
  s <- s / sd(s)
  mean(s[country == "NLD"]) - mean(s[country == "FRA"])
})
round(gap, 2)
  wrong missing    rule 
   0.46    0.28    0.43

The rule splits the missing cells into 7.2% omitted and 1.3% not reached. Here both kinds go with the proportion correct on the answered items (−0.26 and −0.14), unlike PIRLS 2023. The Netherlands–France difference is 0.46 SD with every gap wrong, 0.28 with every gap missing and 0.43 under the split. Without the student weights, these show how far a comparison moves with the scoring, not either country’s performance.

A twist: a table that has already decided

In borges_brazil_residency_2024_cbt, 3,223 applicants to medical residency at a Brazilian hospital took a 120-item computer-based exam (Borges et al., 2024; CC0). Its job is to show a table in which the gaps have already been scored. The source marks some responses “X” (on a computer-based test most likely an item with no option chosen, though the deposit doesn’t say); the IRW script scores an X as 0 and leaves resp_raw empty, so those rows can still be found.

Code 
# Twist: a table that has already scored its gaps. In the source, "X" marks an item
# with no chosen option; the IRW script scores it 0 and blanks resp_raw.
bz <- read.csv("https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v64_0.borges_brazil_residency_2024_cbt/rows?format=csv")
table(resp_raw_empty = is.na(bz$resp_raw), resp = bz$resp)
              resp
resp_raw_empty      0      1
         FALSE 165396 219450
         TRUE    1914      0
Code 
n_x <- tapply(is.na(bz$resp_raw), bz$id, sum)          # X's per applicant
table(cut(n_x, c(-1, 0, 2, 9, 120), labels = c("0", "1-2", "3-9", "10+")))

   0  1-2  3-9  10+ 
2651  546    8   18
Code 
p_right <- tapply(bz$resp, bz$id, mean)
round(tapply(p_right, cut(n_x, c(-1, 0, 2, 120), labels = c("0", "1-2", "3+")), mean), 2)
   0  1-2   3+ 
0.57 0.56 0.30

There are 1,914 such responses, all scored 0. Most applicants (2,651) have none, and 18 have ten or more. Applicants with three or more average 0.30 correct, against 0.57 for those with none. Whether an X is an omit or a not-reached item, the table scores it wrong; problem 2 undoes that.

The exact match in PIRLS 2023 comes from one passage in one part. Does the rule hold where a booklet has two timed parts? TIMSS 2007 grade 8 data from Russia, released with sirt, keep the IEA codes. Each booklet holds four blocks in two parts (Ruddock, O’Sullivan, Arora & Erberber, 2008, Exhibit 2.13), and the code recovers each student’s booklet from the blocks they saw.

Code 
# Sanity check on the rule: TIMSS 2007 grade 8, Russia (sirt::data.timss07.G8.RUS),
# where the IEA codes are kept (6/96 not reached, 9/99 omitted, blank = not given).
library(sirt)
data(data.timss07.G8.RUS)
Tr <- data.timss07.G8.RUS
it <- as.character(Tr$iteminfo$item)
R <- as.matrix(Tr$raw[, it])
tcode <- matrix(ifelse(is.na(R), NA, ifelse(R %in% c(9, 99), "omitted",
                ifelse(R %in% c(6, 96), "not reached", "answered"))), nrow(R))
# Items given to the same students form a block (a mathematics block and the science
# block with the same number go together); the file lists blocks in order M01..M14.
# A few items are blank for a few students, so a block is keyed on the first ten
# students who saw the item.
seen <- !is.na(R)
key <- apply(seen, 2, function(z) paste(head(which(z), 10), collapse = ","))
grp <- match(key, unique(key[colSums(seen) > 0]))
grp[colSums(seen) == 0] <- NA                     # items no student in Russia saw
max(grp, na.rm = TRUE)                            # 14 blocks
[1] 14
Code 
given <- colSums(seen) > 0
c(students = nrow(R), items_given = sum(given),
  saw_every_item = sum(rowSums(seen[, given]) == sum(given)))   # a booklet design
      students    items_given saw_every_item 
          4472            437              0
Code 
subj <- substr(it, 1, 1)
# Booklet k holds blocks k and k+1 (booklet 14 holds 14 and 1). Odd booklets give
# mathematics in part 1 and science in part 2; even booklets the reverse.
rule_all <- truth_all <- whole_all <- nobreak_all <- nobreak_truth <- character()
for (j in which(rowSums(seen) > 0)) {
  bl <- sort(unique(grp[seen[j, ]]))
  k <- if (identical(bl, c(1L, 14L))) 14 else min(bl)
  k2 <- if (k == 14) 1 else k + 1
  blk <- function(s, g) which(grp == g & subj == s)
  first <- if (k %% 2 == 1) "M" else "S"; second <- setdiff(c("M", "S"), first)
  parts <- list(c(blk(first, k), blk(first, k2)), c(blk(second, k), blk(second, k2)))
  # Missing-data choice: an item left blank for this student (not given) is dropped
  # from the sequence.
  parts <- lapply(parts, function(p) p[!is.na(tcode[j, p])])
  for (p in parts) {
    miss <- tcode[j, p] != "answered"
    if (!any(miss)) next
    last <- max(c(0, which(!miss)))
    rule_all <- c(rule_all, classify(miss)[miss])
    whole_all <- c(whole_all, ifelse(seq_along(miss) > last, "not reached", "omitted")[miss])
    truth_all <- c(truth_all, tcode[j, p][miss])
  }
  both <- unlist(parts); miss <- tcode[j, both] != "answered"   # ignore the break
  if (any(miss)) {
    nobreak_all <- c(nobreak_all, classify(miss)[miss])
    nobreak_truth <- c(nobreak_truth, tcode[j, both][miss])
  }
}
table(IEA_code = truth_all, rule = rule_all)
             rule
IEA_code      not reached omitted
  not reached        3804       7
  omitted             123   26399
Code 
# Agreement with the IEA codes, and the share of not-reached cells found, for the
# rule and for two simplifications: every trailing gap not reached, and no break
# between the parts.
agree <- function(pred, truth) c(agree = mean(pred == truth),
  not_reached_found = mean(pred[truth == "not reached"] == "not reached"))
round(rbind(rule = agree(rule_all, truth_all), whole_run = agree(whole_all, truth_all),
            no_part_break = agree(nobreak_all, nobreak_truth)), 3)
              agree not_reached_found
rule          0.996             0.998
whole_run     0.933             1.000
no_part_break 0.923             0.389

No student saw every item, as a booklet design intends. With the parts known, the rule agrees with the IEA codes on 99.6% of missing cells. Calling every trailing gap not reached gets 93.3%, and ignoring the break between parts finds only 39% of the not-reached cells. So the rule needs both the order and the part boundaries.

What each treatment assumes

Each treatment is an assumption. Leaving omits missing assumes they say nothing about \(\theta\); scoring them 0, that the respondent would have got them wrong; a propensity model, that the correlation between answering and ability carries what omitting reveals. Treating not-reached items as not administered assumes the student would have answered them like those who reached them; scoring them wrong counts running out of time as part of the performance.

Some of these assumptions the data can speak to. Whether omitting goes with ability is observable: in PIRLS 2023 South Africa it does, weakly (−0.16 with the plausible value, and a propensity correlation of 0.30), so the simple treatments of omits part company for the learners who omit most (−0.31 left missing, −0.56 scored 0, −0.46 with the propensity model). Where omitting is rare and unrelated to ability, they would barely differ. What an omitted response would have been is never observed, so no fit settles the last step; how far the estimates move across treatments shows how much a conclusion leans on it. For not-reached items, the TIMSS rule (missing when calibrating, wrong when scoring) separates what an item is from what a respondent did, and each half of it rests on one of the assumptions above.

Problems

  1. The factorization. For a Rasch test, show that under MAR with distinct \(\phi\), \(m\) drops out of the likelihood for the item parameters. Then show where the argument breaks when omitting depends on \(\theta\) and the likelihood is marginal over \(\theta\).
  2. Undo the scoring. In borges_brazil_residency_2024_cbt, set resp to missing where resp_raw is empty and fit a Rasch model both ways. Whose estimates move, and how far? Are the X’s of the 18 applicants with ten or more at the end of the exam, or scattered?
  3. Which comparison to report. In PIRLS 2011, the Netherlands–France difference is 0.46, 0.28 or 0.43 SD, depending on the scoring. Which would you report in a comparison of the two countries, and what else would you want to see first?
  4. Formats. In PIRLS 2023, compare the omission rates of the four multiple-choice and eleven constructed-response items. The source’s code 9 means “omitted or invalid”. What might explain the difference, and what does each scoring do to a comparison of formats?
  5. Design a record. For a low-stakes online test with a 20-minute limit, what would you record (timestamps, the order each respondent saw, a “skip” button) so that omitted and not-reached items can be told apart without an order rule?
  6. Challenge (open): what an omit would have been. Whether an omitted item would have been answered correctly is never observed. What assumption lets the Mislevy–Wu model estimate it, and could any design check that assumption? As far as I know, this is not settled.

Stuck, or curious about something this lesson doesn’t cover? Ask Claude about this lesson opens a new conversation that already knows what the lesson is about; add your question at the end. Claude shows a caution notice on any prompt that arrives through a link; that’s expected.

Going further

  • Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. https://doi.org/10.1093/biomet/63.3.581. MCAR, MAR and ignorability.
  • Little, R. J. A., & Rubin, D. B. (2019). Statistical analysis with missing data (3rd ed.). Wiley. https://doi.org/10.1002/9781119482260. A book-length treatment of missing data in general.
  • Seaman, S., Galati, J., Jackson, D., & Carlin, J. (2013). What is meant by “missing at random”? Statistical Science, 28(2). https://doi.org/10.1214/13-STS415. The definitions, stated with care.
  • Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147. A survey of treatments for psychologists.
  • Mislevy, R. J., & Wu, P.-K. (1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing. ETS Research Report Series, 1996(2). https://doi.org/10.1002/j.2333-8504.1996.tb01708.x. Ignorability worked out for item response models, case by case.
  • Mislevy, R. J. (2017). Missing responses in item response modeling. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 2. Statistical tools (pp. 171–194). Chapman and Hall/CRC. https://doi.org/10.1201/b19166-10. A handbook chapter on the same ground.
  • Lord, F. M. (1974). Estimation of latent ability and item parameters when there are omitted responses. Psychometrika, 39(2), 247–264. https://doi.org/10.1007/BF02291471. The case against scoring omits as wrong, and an alternative.
  • De Ayala, R. J., Plake, B. S., & Impara, J. C. (2001). The impact of omitted responses on the accuracy of ability estimation in item response theory. Journal of Educational Measurement, 38(3), 213–234. https://doi.org/10.1111/j.1745-3984.2001.tb01124.x. Treatments of omits compared in simulation.
  • Finch, H. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225–245. https://doi.org/10.1111/j.1745-3984.2008.00062.x. Treatments of missing data compared for item parameters.
  • Holman, R., & Glas, C. A. W. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1–17. https://doi.org/10.1348/000711005X47168. Missingness as a second latent variable.
  • Rose, N., von Davier, M., & Xu, X. (2010). Modeling nonignorable missing data with item response theory (IRT). ETS Research Report Series, 2010(1). https://doi.org/10.1002/j.2333-8504.2010.tb02218.x. Propensity models for omitted responses.
  • Rose, N., von Davier, M., & Nagengast, B. (2017). Modeling omitted and not-reached items in IRT models. Psychometrika, 82(3), 795–819. https://doi.org/10.1007/s11336-016-9544-7. How the models relate, the Mislevy–Wu model among them.
  • Pohl, S., Gräfe, L., & Rose, N. (2014). Dealing with omitted and not-reached items in competence tests. Educational and Psychological Measurement, 74(3), 423–452. https://doi.org/10.1177/0013164413504926. Treatments compared on the NEPS competence tests.
  • Köhler, C., Pohl, S., & Carstensen, C. H. (2015). Taking the missing propensity into account when estimating competence scores. Educational and Psychological Measurement, 75(5), 850–874. https://doi.org/10.1177/0013164414561785. Whether a propensity model changes competence scores.
  • Robitzsch, A. (2021). On the treatment of missing item responses in educational large-scale assessment data. European Journal of Investigation in Health, Psychology and Education, 11(4), 1653–1687. https://doi.org/10.3390/ejihpe11040117. A review for large-scale assessments.
  • Debeer, D., Janssen, R., & De Boeck, P. (2017). Modeling skipped and not-reached items using IRTrees. Journal of Educational Measurement, 54(3), 333–363. https://doi.org/10.1111/jedm.12147. Skips and not-reached items as nodes of a tree.
  • Oshima, T. C. (1994). The effect of speededness on parameter estimation in item response theory. Journal of Educational Measurement, 31(3), 200–219. https://doi.org/10.1111/j.1745-3984.1994.tb00443.x. What a time limit does to late items’ parameters.
  • Bolt, D. M., Cohen, A. S., & Wollack, J. A. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39(4), 331–348. https://doi.org/10.1111/j.1745-3984.2002.tb01146.x. Speeded respondents as a latent class.
  • Glas, C. A. W., & Pimentel, J. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68(6), 907–922. https://doi.org/10.1177/0013164408315262. A propensity model for not-reached items.
  • van der Linden, W. J. (2018). Test speededness and time limits. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 3. Applications (ch. 12, pp. 249–266). Chapman and Hall/CRC. https://www.taylorfrancis.com/chapters/edit/10.1201/9781315117430-12. Speededness and time limits in full.
  • Foy, P., Galia, J., & Li, I. (2008). Scaling the data from the TIMSS 2007 mathematics and science assessments. In J. F. Olson, M. O. Martin, & I. V. S. Mullis (Eds.), TIMSS 2007 technical report (ch. 11). TIMSS & PIRLS International Study Center, Boston College. https://timssandpirls.bc.edu/TIMSS2007/PDF/T07_TR_Chapter11.pdf. The not-reached rule and how TIMSS treats omitted and not-reached items (§11.3.3).
  • Ruddock, G. J., O’Sullivan, C. Y., Arora, A., & Erberber, E. (2008). Developing the TIMSS 2007 mathematics and science assessments and scoring guides. In J. F. Olson, M. O. Martin, & I. V. S. Mullis (Eds.), TIMSS 2007 technical report (ch. 2). https://timssandpirls.bc.edu/TIMSS2007/PDF/T07_TR_Chapter2.pdf. The booklet design used in the check (Exhibit 2.13).
  • Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. https://doi.org/10.1007/BF02296272. The partial credit model fitted to the PIRLS items.
  • Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443–459. https://doi.org/10.1007/BF02293801. The marginal likelihood mirt maximizes.
  • Enders, C. K., & Bandalos, D. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8(3), 430–457. https://doi.org/10.1207/S15328007SEM0803_5. FIML against deletion.
  • Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6). https://doi.org/10.18637/jss.v048.i06. Every fit in this lesson, including the two-dimensional propensity model.

Data sources

Data from the Item Response Warehouse (each table name links to its IRW page); references from IRW biblio.

For instructors

  • Source: no EDUC 252 class covers missing responses; the lesson is new (#153). The data audit in outlines/missing-responses.md records how the IRW represents missing responses and which tables keep the codes.
  • Timing: the core ideas and widgets take about 60 minutes; the PIRLS 2023 analysis works as a 30-minute lab. Every fit takes seconds in local R, the propensity model about ten.
  • Data: the PIRLS 2023 check downloads the source file (4.5 MB, CC BY 4.0) from the University of Pretoria’s repository. Only the check and the plausible value depend on it; everything else uses the IRW table alone. The TIMSS 2007 check needs sirt.
  • Held back: rapid guessing as a missing-data screen (Mixture models); plausible values with background conditioning, which is how TIMSS and PIRLS score students; missingness by design in adaptive testing (Item banks and adaptive testing). ENEM tables carry each candidate’s item order, but blanks are rare there (a blank earns what a wrong answer earns), so they stay in the outline.
  • Solutions: held back in solutions/missing-responses.qmd (not published).