Mixture models: when respondents differ in kind

Module: Beyond the standard models
Tranche: extension (beyond a first course)
Before this: Guessing and priors · Person fit: does a respondent’s pattern fit the model? · From construct map to items: building an instrument · Models for polytomous responses
Builds toward: none
Picks up threads: Alpha is a lower bound on reliability (from Classical test theory and reliability); Keying has to be checked; the data can’t always tell (from Classical test theory and reliability); The sum score is sufficient under the Rasch model (from The Rasch model); Item comparisons don’t depend on who took the items (from The Rasch model); Misfit as overfit or underfit (outfit and infit) (from The Rasch model); Check keying in the documentation and in the data (from Item response data and the IRW); The IMV: predictive gain on a portable scale (from Model fit and out-of-sample prediction); Wording direction can act as a second dimension (from From construct map to items: building an instrument); Where chance sets the floor, not knowing means guessing (from Guessing and priors); Ordered latent classes can look like a continuum cut into pieces (from Constructs and construct maps); Reliability, like any correlation, depends on how much the population varies (from Classical test theory and reliability); Acquiescence and other response styles are shared by every item; alpha counts them as signal (from From construct map to items: building an instrument); A misfitting pattern says the model doesn’t describe this respondent, not why (from Person fit: does a respondent’s pattern fit the model?)
IRW tables: kay_2025_antonyms, su_2024_pss14, su_2024_phq9, test_taking_much_2025_mr
From EDUC 252: n/a
Status: draft
Authorship: Written largely by Claude (Anthropic), from Ben Domingue’s EDUC 252 materials and under Ben’s direction. Ben reviews each lesson before it is marked done.
Comments: open an issue on GitHub about this lesson.

What this is for

Every model so far has assumed one measurement model for every respondent. What if some respondents answer a different question, or none at all? On an online extraversion survey analysed below, about half the respondents from one platform answer as if the content of the items didn’t matter, and their sum scores are exactly as high as everyone else’s. A mixture model lets respondents belong to latent classes, each with its own measurement model, and asks the data who is in which. Guessing and priors fitted the simplest one. This lesson takes the idea to inattentive responding and response styles, and asks when a class is a kind of respondent and when it merely patches the model.

Goals

By the end of this lesson you should be able to:

  • write a pattern’s likelihood under a mixture and compute posterior class probabilities;
  • say when two respondents with the same sum score can have different posteriors;
  • compute the usual screens for careless responding, and explain why they disagree;
  • explain why a content-free class shows on a balanced scale and hides in the trait on an all-forward one;
  • fit mixture Rasch and mixed partial credit models in mirt, choose the number of classes, and check a class against outside evidence.

Core ideas

One population, or several?

Suppose each respondent belongs to one of \(K\) latent classes, with shares \(\pi_1, \dots, \pi_K\) that sum to 1, and each class \(k\) has its own measurement model. In the mixture Rasch model (Rost, 1990; Mislevy & Verhelst, 1990), class \(k\) has its own item difficulties \(b_{ik}\) and its own distribution of \(\theta\). For a respondent whose class we don’t know, the probability of a pattern \(\mathbf{x}\) is a weighted sum over classes, and Bayes’ rule turns it around, as the EAP of Estimating abilities did with a prior over \(\theta\): \[\Pr(\mathbf{x}) = \sum_{k=1}^K \pi_k \Pr(\mathbf{x} \mid k), \qquad \Pr(k \mid \mathbf{x}) = \frac{\pi_k \Pr(\mathbf{x} \mid k)}{\sum_{m} \pi_m \Pr(\mathbf{x} \mid m)}.\]

Recall from Guessing and priors: in the “safety valve” of Xiao et al. (2026), a share \(\pi\) of respondents follow the Rasch model and the rest answer at chance; on roar_lexical, \(\hat\pi\) was 0.87. One class’s model is fixed in advance.

Here neither class is fixed. On a survey, the second class might be a content-free class: respondents who agree (or disagree) whatever an item says.

Try it: whose pattern is it?

Ten keyed items, five worded forward (F1–F5) and five reversed (R1–R5), with difficulties from −1 to 1 in each half. Engaged respondents follow the Rasch model, \(\theta\) standard normal; content-free respondents agree with any item with the probability you set. Tick the items scored 1.

At the start, the five forward items are scored 1: a sum of 5, and a posterior of 0.997 for the content-free class. Score F1, F2, F3, R1 and R2 instead: the same sum, but a posterior of 0.065. An alternating pattern (F1, F3, F5, R2, R4) is unlikely under both classes, and its posterior (0.58) stays near the prior.

Recall from The Rasch model: the sum score is sufficient for \(\theta\), and item comparisons don’t depend on who took the items (specific objectivity).

Both hold within a class, not between classes. Why can the pattern carry the class when the sum can’t? If two classes’ difficulties differ only by a constant, the class posterior depends on the pattern only through the sum (first Go deeper callout): the classes are levels of one continuum. Only when the difficulty profiles cross does the pattern carry the class, and then the classes differ in kind. If you’ve done Latent classes and cognitive diagnosis, classes there were fixed by the attributes mastered; here a class is defined by the model a respondent follows.

Assumptions: a mixture Rasch model with classes \(k = 1, \dots, K\), shares \(\pi_k\), difficulties \(b_{ik}\) and ability densities \(g_k(\theta)\); local independence within a class.

Within a class. Writing \(r = \sum_i x_i\), \[\Pr(\mathbf{x} \mid \theta, k) = \prod_i \frac{\exp\{x_i(\theta - b_{ik})\}}{1 + \exp(\theta - b_{ik})} = \exp\Big(-\sum_i x_i b_{ik}\Big)\,\frac{\exp(r\theta)}{\prod_i \{1 + \exp(\theta - b_{ik})\}}.\] Integrating over \(g_k\) gives \(\Pr(\mathbf{x} \mid k) = \exp(-\sum_i x_i b_{ik})\,h_k(r)\), where \(h_k(r)\) depends on the pattern only through \(r\) (Rost, 1990, writes the same factorization with each class’s score distribution).

The posterior odds. For two classes, \[\log\frac{\Pr(1 \mid \mathbf{x})}{\Pr(2 \mid \mathbf{x})} = \log\frac{\pi_1 h_1(r)}{\pi_2 h_2(r)} + \sum_i x_i\,(b_{i2} - b_{i1}).\] The first term is a function of the sum. The second is where the pattern comes in, each item weighted by the difference between the classes’ difficulties.

Levels. If \(b_{i2} - b_{i1} = \delta\) for every item, the second term is \(r\delta\), a function of the sum. Respondents with the same sum have the same posterior, whatever their patterns.

Kinds. If the differences vary across items (the profiles cross), two patterns with the same \(r\) get different posteriors, and the items on which the classes disagree most weigh most. In the widget the content-free class behaves like difficulties of about −1.4 on the forward items and +1.4 on the reversed ones, so a forward item scored 1 pushes toward that class and a reversed item scored 1 pushes away from it.

Screening one respondent at a time

The usual first pass is person-level screens (Meade & Craig, 2012; Curran, 2016): speed, with a cut set by how long reading takes (Huang et al., 2012; Wise & Kong, 2005); sameness, the longest run of one option or the within-person SD (Johnson, 2005); inconsistency between two halves of a scale; oddness, the Mahalanobis distance or person fit; and direct checks, instructed-response and nonsense items (Oppenheimer et al., 2009; Maniaci & Rogge, 2014).

Recall from Person fit: \(l_z\) (mirt’s Zh) standardizes a pattern’s log likelihood at \(\hat\theta\); below 0 is underfit, above 0 overfit. Misfit says the model doesn’t describe the respondent, not why.

A mixture proposes the model that does. But first, do the screens agree?

Try it: which screen sees which?

2,000 simulated respondents answer ten five-point items, five reversed. Beside the engaged majority are three kinds, each a share you choose: random answering, one option throughout, and engaged but fast. Speed flags anyone under the cut; the other screens flag their worst 5%.

No screen sees every kind: sameness flags every straight line and no random responder; even–odd and Mahalanobis catch some random responders and almost no fast readers; speed flags about three-quarters of all three fast kinds, readers included. Careless responding isn’t one behaviour, and each screen sees one of them (Niessen et al., 2016). A straight line also overfits: it is more predictable than the model expects.

The content-free class

What does a content-free class do to a scale? That depends on keying.

Recall from From construct map to items: acquiescence is shared by every item, so on an all-forward scale alpha counts it as signal; on a balanced scale it cancels in the sum. Reversed items can form a wording factor.

Recall from Classical test theory and reliability: alpha is a lower bound on reliability only when errors are uncorrelated; correlated errors can push it above reliability.

That widget gave every respondent a little acquiescence. Here a few respondents have all of it.

Try it: balanced or all forward?

800 simulated respondents answer ten five-point items. Engaged respondents follow one trait; a content-free class either gives one option throughout (drawn at random per respondent) or agrees with everything (4 or 5).

On the balanced scale every straight line scores exactly 30, the midpoint. Alpha falls from 0.90 without them to 0.85, and the second eigenvalue rises from 0.62 to 2.47: a wording factor made of respondents. Switch to all forward and the class spreads across the whole range, with alpha 1.00 inside it, and the scale’s alpha rises to 0.93. “Agree with everything” on the all-forward scale puts the class near the top (a mean of 45), where it passes for respondents high on the trait. Balanced keying is what makes a content-free class visible (Woods, 2006; Kam & Meyer, 2015; Arias et al., 2020). Weijters, Baumgartner and Schillewaert (2013) count careless responding among the wording factor’s causes; Jin, Chen and Wang (2018), Ulitzsch, Yildirim-Erbasli et al. (2022) and van Laar and Braeken (2022) model the class directly.

Assumptions: two classes with shares \(\pi\) (engaged) and \(1 - \pi\), mean vectors \(\boldsymbol\mu_1, \boldsymbol\mu_0\) and covariance matrices \(\Sigma_1, \Sigma_0\) of the keyed responses.

The mixture’s covariance is the average within-class covariance plus the covariance of the class means: \[\Sigma = \pi\Sigma_1 + (1 - \pi)\Sigma_0 + \pi(1 - \pi)(\boldsymbol\mu_1 - \boldsymbol\mu_0)(\boldsymbol\mu_1 - \boldsymbol\mu_0)^\top.\]

A straight-line class. On a 1–5 scale, a respondent who chooses option \(o\) everywhere has keyed responses \(o\) on forward items and \(6 - o\) on reversed ones: \(3 + (o - 3)\mathbf{v}\), with \(v_i = +1\) for forward and \(-1\) for reversed items. So \(\Sigma_0 = \text{Var}(o)\,\mathbf{v}\mathbf{v}^\top\), a rank-one term along the keying vector. On an all-forward scale \(\mathbf{v}\) is all ones and lies along the trait’s first factor: covariance is added everywhere and alpha rises. On a balanced scale the term adds covariance within each half and subtracts it across halves, which is a wording factor, and the sum score gets no variance from it (\(\mathbf{1}^\top\mathbf{v} = 0\)), so alpha falls.

An agree-with-everything class has its own mean: on a balanced scale \(\boldsymbol\mu_0 - \boldsymbol\mu_1\) points roughly along \(\mathbf{v}\), and the last term is again close to rank one along the keying vector. This algebra is behind the Su alphas and eigenvalues below.

Response styles as classes

Extreme and midpoint responding are tendencies, not lapses: fairly stable within a respondent and shared by every item (Baumgartner & Steenkamp, 2001). Can a mixture find them?

Recall from Polytomous responses: under the partial credit model, category \(k\) is preferred to \(k - 1\) with log odds \(\theta - b_k\), \(b_k\) being the step difficulty.

In the mixed partial credit model (Rost, 1991) each class has its own steps: squeezed steps send a class to the ends, spread steps keep it in the middle. Both kinds have been reported on questionnaires (Eid & Rauber, 2000; Hernández, Drasgow & González-Romá, 2004; Austin, Deary & Egan, 2006; Wetzel, Carstensen & Böhnke, 2013).

Try it: levels or styles?

One item scored 0–4 under the partial credit model, with steps at −1.5, −0.5, 0.5 and 1.5 for class A. Class B’s steps are A’s times a spread, and its mean \(\theta\) can shift. Bars show category use, averaged over \(\theta\).

At a spread of 0.3 and no shift, both classes average 2.00, but B puts 0.47 of its responses in the end categories against A’s 0.24: an extreme style. With the spread at 1 and a shift, B differs only in level. Combine the two and B uses the ends more and sits lower. So ask of any style class: style, or trait? If you’ve done IRTree models, they separate the two by modelling the choice of an end category as its own node (Böckenholt, 2012).

How many classes, and are they real?

Another class always raises the likelihood. BIC is one way to choose \(K\) (Li et al., 2009; Nylund et al., 2007).

Recall from Model fit and out-of-sample prediction: the IMV scores predictions of held-out responses on a portable scale (Domingue et al., 2024).

Held-out responses are the other route: a real class should predict them. Two practical matters follow every fit. Label switching: classes come out in arbitrary order (Stephens, 2000), so name them by profile, not number. Local maxima: a mixture’s likelihood can have several peaks, so fit from several random starts (mirt’s nruns argument; Chalmers, 2012).

Recall from What is a construct?: ordered latent classes close together look very much like a continuum cut into pieces.

The converse holds too. A skewed continuum, or an item model wrong for everyone, can be fitted better by two classes than one, and the classes then describe the model’s shortfall, not kinds of people (Bauer & Curran, 2003; Lubke & Muthén, 2005). So naming a class assumes more than the fit shows, and outside evidence is what backs the name.

When responses can’t tell, times can

On a test with no chance floor, a disengaged respondent and an engaged one who finds every item too hard may both get everything wrong. No mixture of response models can separate identical patterns. What else did the respondent give us? Their time: a disengaged respondent tends to answer quickly and an engaged low scorer slowly (Wang & Xu, 2015; Ulitzsch, Pohl et al., 2022). The simplest version gives each class its own distribution of mean log time per item. If you’ve done Response time and the speed-accuracy tradeoff, it modelled log time item by item; the full joint model is beyond this lesson.

Simulate

The code below simulates 1,000 respondents on ten balanced items (80% Rasch, 20% agreeing with any item with probability 0.8), fits one and two classes in mirt, compares the estimates with the truth, and cross-tabulates three screens against the true class. The first run takes 20–30 seconds while R and mirt load, and the mixture up to a minute more.

With the seed as written, the classes’ mean sum scores are 5.10 and 5.04. The mixture estimates the content-free share at 0.20 (truth 0.21), with forward difficulties of −1.2 to −1.6 and reversed ones of 1.1 to 1.6 (truth ±1.39, the logit of 0.8), and puts 152 of the 208 content-free respondents in their class, with 27 of the 792 engaged ones. Speed flags 42% of the content-free respondents, sameness 64%, and the Mahalanobis distance 10%, no more than among the engaged. Try p_agree <- 0.5 (random answering) or reversed <- integer(0) (all forward), and other seeds, where the class numbers may swap.

mirt’s d is an easiness, so the code converts with \(b = -d\).

Download this code to run it with more respondents in your own R session.

With real data

Antonyms on three platforms

Our main example, kay_2025_antonyms, is a data-quality probe (Kay, 2025; survey and data on OSF, CC BY 4.0): 1,200 respondents from Connect, Prolific and MTurk rated 29 pairs of statements, mostly opposites such as “I am an extravert” and “I am an introvert” (wording from the survey), from −3 to +3, alongside four instructed-response items. Its job: a mixture finding a content-free class unaided. We key ten extraversion items, five each way, and score agreement (“slightly agree” or more). Download the code to run every chunk yourself; it needs mirt (Chalmers, 2012) and psych (Revelle, 2026).

Code 
library(mirt)
long2wide <- function(df) as.data.frame(tapply(df$resp, list(df$id, df$item), function(x) x[1]))
kay_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse_5:v5_0.kay_2025_antonyms/rows?format=csv"
kay <- read.csv(kay_url)
# resp is the raw rating from -3 (strongly disagree) to +3 (strongly agree). Items
# come in pairs; the _x/_r suffix only marks the first and second member of a pair,
# not the direction of any trait, so we key items from their wording. No waves.
W <- long2wide(kay)
person <- kay[!duplicated(kay$id), ]
person <- person[match(rownames(W), person$id), ]
platform <- ifelse(grepl("MTurk", person$cov_source), "MTurk", "Connect/Prolific")
checks_failed <- person$cov_att_chk        # instructed-response items failed, of 4
table(source = person$cov_source, checks_failed)
           checks_failed
source        0   1   2   3   4
  Connect    91   8   1   0   0
  MTurk_400 260  70  34  19  17
  MTurk_600 445  82  30  22  21
  Prolific   90   9   0   1   0
Code 
# The last pair: "I am an extravert" / "I am an introvert". Their raw correlation:
pair_r <- function(ix) round(cor(W$extr_057_x[ix], W$extr_058_r[ix]), 2)
c(connect_prolific = pair_r(platform != "MTurk"),
  mturk_failed_at_most_one = pair_r(platform == "MTurk" & checks_failed <= 1))
        connect_prolific mturk_failed_at_most_one 
                   -0.80                     0.18
Code 
# Ten extraversion items, five worded each way. Agreeing (slightly or more) with a
# forward item, or not agreeing with a reversed one, scores 1: higher = more
# extraverted.
forward  <- c("tlk1_001_x", "tlk2_003_x", "cntr_006_r", "frnd_031_x", "extr_057_x")
reversed <- c("tlk1_002_r", "tlk2_004_r", "cntr_005_x", "frnd_032_r", "extr_058_r")
agree <- (W[, c(forward, reversed)] >= 1) * 1
X <- cbind(agree[, forward], 1 - agree[, reversed])
colnames(X) <- c(paste0("F", 1:5), paste0("R", 1:5))
round(colMeans(X), 2)
  F1   F2   F3   F4   F5   R1   R2   R3   R4   R5 
0.70 0.72 0.55 0.75 0.66 0.24 0.38 0.27 0.36 0.31
Code 
# In the whole sample, the average correlation of a keyed forward item with a keyed
# reversed one (they should correlate positively):
round(mean(cor(X)[1:5, 6:10]), 2)
[1] -0.25

On Connect and Prolific the extravert and introvert statements correlate −0.80; on MTurk, among respondents who failed at most one check, +0.18.

Code 
# One class (the Rasch model), then two and three classes, each with its own item
# difficulties, mean and variance of theta, and a share pi_k. mirt fits mixtures
# with multipleGroup(..., dentype = "mixture-K"). Each fit starts from random values;
# code/mixture-models-precompute.R refits from ten starts and always reaches the
# same maximum. About a minute and a half.
set.seed(1)
kay1 <- mirt(X, 1, itemtype = "Rasch", verbose = FALSE)
kay2 <- multipleGroup(X, 1, itemtype = "Rasch", dentype = "mixture-2", verbose = FALSE,
                      technical = list(NCYCLES = 3000))
kay3 <- multipleGroup(X, 1, itemtype = "Rasch", dentype = "mixture-3", verbose = FALSE,
                      technical = list(NCYCLES = 3000))
fits <- list(one = kay1, two = kay2, three = kay3)
data.frame(classes = 1:3,
           logLik = round(sapply(fits, extract.mirt, "logLik")),
           parameters = sapply(fits, extract.mirt, "nest"),
           BIC = round(sapply(fits, extract.mirt, "BIC")))
classes logLik parameters BIC
one 1 -7369 11 14816
two 2 -6088 23 12340
three 3 -5934 35 12116
Code 
# The two classes' profiles. mirt's d is an easiness, so b = -d. Which class is
# "first" depends on the start (label switching), so we name them by profile: the
# class whose theta barely varies is the one in which content doesn't matter.
cf <- coef(kay2, simplify = TRUE)
theta_var <- sapply(cf, function(g) g$cov[1, 1])
cfree <- which.min(theta_var)
profiles <- sapply(cf, function(g) -g$items[, "d"])
colnames(profiles) <- ifelse(seq_along(cf) == cfree, "content_free", "engaged")
round(profiles, 1)
   content_free engaged
F1         -3.6     0.3
F2         -3.3     0.1
F3         -1.2     0.8
F4         -3.3    -0.1
F5         -2.8     0.6
R1          3.4     0.3
R2          2.4    -0.8
R3          2.9     0.1
R4          2.4    -0.7
R5          2.5    -0.2
Code 
round(c(share_content_free = unname(extract.mirt(kay2, "pis")[cfree]),
        theta_var_content_free = unname(theta_var[cfree]), theta_var_engaged = unname(theta_var[-cfree])), 3)
    share_content_free theta_var_content_free      theta_var_engaged 
                 0.498                  0.045                  0.729
Code 
post <- fscores(kay2, method = "classify", verbose = FALSE)
p_cf <- post[, grep("^CLASS", colnames(post))][, cfree]   # posterior of the content-free class
in_cf <- p_cf > 0.5
c(assigned = sum(in_cf), share_sure = round(mean(p_cf > 0.9 | p_cf < 0.1), 2))
  assigned share_sure 
    598.00       0.88

BIC falls from 14,816 with one class to 12,340 with two and 12,116 with three. In the two-class solution, one class (a share of 0.50) finds the forward items easy (difficulties −3.6 to −1.2) and the reversed ones hard (2.4 to 3.4), with almost no spread in \(\theta\) (variance 0.045): agreement, whatever the statement. The other class’s difficulties lie between −0.8 and 0.8. The profiles cross, and 88% of the posteriors are above 0.9 or below 0.1. Who is in the content-free class? The model never saw the platform or the check items.

Code 
# Outside evidence the model never saw. By platform:
table(platform, content_free = in_cf)
                  content_free
platform           FALSE TRUE
  Connect/Prolific   196    4
  MTurk              406  594
Code 
# By instructed-response items passed (all four) or not:
round(prop.table(table(passed_all_four = checks_failed == 0, content_free = in_cf), 1), 2)
               content_free
passed_all_four FALSE TRUE
          FALSE  0.33 0.67
          TRUE   0.56 0.44
Code 
round(tapply(in_cf[checks_failed == 0], platform[checks_failed == 0], mean), 2)
Connect/Prolific            MTurk 
            0.02             0.54
Code 
# Agreement (slightly or more) with either gibberish item ("I am ffhjhl", "I am sqnmmp"):
gibberish <- W$gibb_045_x >= 1 | W$gibb_046_r >= 1
round(tapply(gibberish, ifelse(in_cf, "content_free", "engaged"), mean), 2)
content_free      engaged 
        0.90         0.28
Code 
# The raw correlation of each antonym pair outside the scale (leaving out the
# extraversion pairs, the gibberish pair, and three pairs that aren't opposites:
# blue, pets, sick), within each class. Opposites should correlate negatively.
pairs <- setdiff(unique(sub("_\\d+_[xr]$", "", names(W))),
                 c("tlk1", "tlk2", "cntr", "frnd", "extr", "gibb", "blue", "pets", "sick"))
pair_cor <- sapply(pairs, function(p) {
  cols <- grep(paste0("^", p, "_"), names(W))
  c(content_free = cor(W[in_cf, cols[1]], W[in_cf, cols[2]]),
    engaged = cor(W[!in_cf, cols[1]], W[!in_cf, cols[2]]))
})
c(pairs = length(pairs), round(apply(pair_cor, 1, median), 2))
       pairs content_free      engaged 
       20.00         0.28        -0.17
Code 
# The keyed sum score in each class:
round(tapply(rowSums(X), ifelse(in_cf, "content_free", "engaged"), mean), 2)
content_free      engaged 
        4.95         4.95

The class holds 4 of the 200 Connect and Prolific respondents and 594 of the 1,000 from MTurk; 44% of those who passed every check and 67% of the rest. 90% of the class agree with a nonsense statement (“I am ffhjhl” or its partner, wording from the survey), against 28% of the others. On the 20 antonym pairs the model never saw, the median raw correlation is +0.28 within the class and −0.17 outside it. And both classes have a mean keyed sum of 4.95: only the pattern sees the class.

Code 
# Held-out responses (five folds over responses): the IMV of each model over another,
# from code/mixture-models-precompute.R (about 15 minutes), with BIC for four classes.
kay_cv <- read.csv("data/mixture-models-kay.csv")
print(format(kay_cv[, c("comparison", "value")], scientific = FALSE, drop0trailing = TRUE), right = FALSE)
  comparison                            value       
1 2 classes over Rasch                      0.1737  
2 3 classes over 2                          0.01618 
3 2PL over Rasch                            0.1634  
4 2 classes over 2PL                        0.008842
5 BIC, 4 classes                        12110       
6 10 starts, 2 classes: distinct maxima     1       
7 10 starts, 3 classes: distinct maxima     1

On held-out responses, two classes gain an IMV of 0.17 over the Rasch model, more than the 0.1 benchmark for the Rasch model over item proportions (Model fit and out-of-sample prediction). A third class adds 0.016; ten random starts reach one maximum. The 2PL alone gains 0.16: with keyed forward and reversed items correlating −0.25 on average (first chunk), its single dimension becomes agreement rather than extraversion. The mixture adds 0.0088 over it.

The same students on two keying designs

Our contrast is a pair of tables from a mandatory online mental-health screening of students at a Chinese medical university (Su et al., 2024), with a time for every response: the balanced Perceived Stress Scale (Cohen, Kamarck & Mermelstein, 1983) in su_2024_pss14 and the all-forward PHQ-9 (Kroenke, Spitzer & Williams, 2001) in su_2024_phq9. Their one job: the same respondents on both keying designs.

Recall from Item response data and the IRW: check keying in the documentation and in the data. The landing page doesn’t say, but the deposit’s total equals the sum of the codes minus one, so the stored PSS codes are already keyed (higher = more stress).

Code 
# Su et al. (2024): a mental-health screening of students at a medical university,
# with a response time (seconds) for every item. Items were given in a fixed order.
su_pss <- read.csv("https://redivis.com/api/v1/tables/datapages.item_response_warehouse_5:v5_0.su_2024_pss14/rows?format=csv")
su_phq <- read.csv("https://redivis.com/api/v1/tables/datapages.item_response_warehouse_5:v5_0.su_2024_phq9/rows?format=csv")
wide <- function(df, v) {
  num <- as.integer(sub(".*_", "", df$item))
  m <- tapply(df[[v]], list(df$id, num), function(x) x[1])
  m[order(as.numeric(rownames(m))), ]
}
P  <- wide(su_pss, "resp"); PT <- wide(su_pss, "rt")
Q  <- wide(su_phq, "resp"); QT <- wide(su_phq, "rt")
Q  <- Q[rownames(P), ]; QT <- QT[rownames(P), ]
c(respondents = nrow(P), complete = sum(complete.cases(P)), phq_matched = sum(complete.cases(Q)))
respondents    complete phq_matched 
      24292       24292       24292
Code 
# PSS codes are 1-5 and arrive already keyed (higher = more stress): the deposit's
# total equals the sum of (code - 1) for every respondent. The seven positively
# worded items are 4, 5, 6, 7, 9, 10 and 13. PHQ-9 codes are 0-3, all worded forward.
positive <- c(4, 5, 6, 7, 9, 10, 13)
# Each respondent's median time per item on the PSS:
med_t <- apply(PT, 1, median)
round(quantile(med_t, c(0.1, 0.17, 0.25, 0.5, 0.75)), 2)
 10%  17%  25%  50%  75% 
0.58 1.02 2.04 3.40 4.60
Code 
fast <- med_t < 1
c(fast = sum(fast), share = round(mean(fast), 2))
   fast   share 
4078.00    0.17

4,078 respondents (17%) take under a second per item; the median takes 3.4 seconds.

Code 
# The most common pattern in the fast group, in keyed codes:
pattern <- apply(P, 1, paste, collapse = "")
top <- names(sort(table(pattern[fast]), decreasing = TRUE))[1]
c(pattern = top, share_of_fast = round(mean(pattern[fast] == top), 2),
  keyed_score = sum(as.integer(strsplit(top, "")[[1]]) - 1))
         pattern    share_of_fast      keyed_score 
"11155551551151"           "0.31"             "28"
Code 
# Un-keying the positive items gives what was clicked: the same option on every screen.
screen <- P; screen[, positive] <- 6 - screen[, positive]
screen[which(pattern == top)[1], ]
 1  2  3  4  5  6  7  8  9 10 11 12 13 14 
 1  1  1  1  1  1  1  1  1  1  1  1  1  1
Code 
alpha <- function(M) psych::alpha(M, check.keys = FALSE, warnings = FALSE)$total$raw_alpha
# Alpha of each scale among fast and slower respondents, and all together:
round(rbind(PSS_balanced = c(fast = alpha(P[fast, ]), slower = alpha(P[!fast, ]), all = alpha(P)),
            PHQ9_forward = c(fast = alpha(Q[fast, ]), slower = alpha(Q[!fast, ]), all = alpha(Q))), 2)
             fast slower  all
PSS_balanced 0.00   0.85 0.79
PHQ9_forward 0.93   0.87 0.88
Code 
# The fast group on the PHQ-9: all nine answered "not at all" (0)?
round(tapply(rowSums(Q) == 0, ifelse(fast, "fast", "slower"), mean), 2)
  fast slower 
  0.52   0.29
Code 
# The PSS's largest eigenvalues, with and without the fast group:
round(rbind(everyone = eigen(cor(P))$values[1:3],
            without_fast = eigen(cor(P[!fast, ]))$values[1:3]), 1)
             [,1] [,2] [,3]
everyone      5.3  3.7  0.8
without_fast  5.3  2.3  1.0
Code 
# Respondents who are fast on one scale are fast on the other:
round(cor(log(med_t), log(apply(QT, 1, median))), 2)
[1] 0.59

A third of the fast group gave one keyed pattern; un-keyed, it is the same option on every screen, and it scores 28 of 56, the midpoint. On the balanced PSS the fast group’s alpha is 0.00, against 0.85 for slower respondents. On the all-forward PHQ-9 the same respondents’ alpha is 0.93, above the slower respondents’ 0.87, and 52% of them answered “not at all” to all nine items (29% of the rest). The PSS’s second eigenvalue is 3.7 with everyone and 2.3 without the fast group. These are the widget’s three effects, and in a screening they matter: a straight line on the PHQ-9 reads as no symptoms. And since reliability belongs to a population (Classical test theory and reliability), including the fast respondents moves the PSS’s alpha from 0.85 to 0.79.

Code 
# Five person-level screens on a random 5,000, each flagging its worst 5%.
set.seed(5)
ix <- sample(nrow(P), 5000)
Ps <- P[ix, ]; Ss <- screen[ix, ]; ts <- med_t[ix]
long_string <- apply(Ss, 1, function(r) max(rle(r)$lengths))   # in the order shown
person_sd   <- apply(Ss, 1, sd)
# Inconsistency: split each subscale's items into alternate halves and add the
# absolute differences between the halves' means.
stress <- setdiff(1:14, positive)
halves <- function(r, s) abs(mean(r[s[c(TRUE, FALSE)]]) - mean(r[s[c(FALSE, TRUE)]]))
even_odd <- apply(Ps, 1, function(r) halves(r, stress) + halves(r, positive))
mahal <- mahalanobis(Ps, colMeans(Ps), cov(Ps))
# Person fit: Z_h (l_z) under a graded response model, at EAP estimates.
grm <- mirt(as.data.frame(Ps - 1), 1, itemtype = "graded", verbose = FALSE)
zh <- personfit(grm)$Zh
flags <- data.frame(speed = ts <= quantile(ts, 0.05),
                    long_string = long_string >= quantile(long_string, 0.95),
                    even_odd = even_odd >= quantile(even_odd, 0.95),
                    mahalanobis = mahal >= quantile(mahal, 0.95),
                    Zh = zh <= quantile(zh, 0.05))
# How many respondents each pair of screens both flag (the diagonal: each screen's
# own count; ties at the cut let some screens flag more than 5%):
crossprod(as.matrix(flags) * 1)
            speed long_string even_odd mahalanobis  Zh
speed         251          83        0           7   2
long_string    83         311        0           2   2
even_odd        0           0      275          63  32
mahalanobis     7           2       63         250 129
Zh              2           2       32         129 250
Code 
round(cor(data.frame(slowness = log(ts), long_string, even_odd, mahal, minus_Zh = -zh),
          method = "spearman"), 2)
            slowness long_string even_odd mahal minus_Zh
slowness        1.00       -0.53     0.32  0.21     0.13
long_string    -0.53        1.00    -0.43 -0.35    -0.19
even_odd        0.32       -0.43     1.00  0.45     0.30
mahal           0.21       -0.35     0.45  1.00     0.84
minus_Zh        0.13       -0.19     0.30  0.84     1.00
Code 
# Respondents who gave one option on every screen, against everyone:
round(c(all_same_n = sum(person_sd == 0), all_same_Zh = median(zh[person_sd == 0]),
        everyone_Zh = median(zh)), 2)
 all_same_n all_same_Zh everyone_Zh 
     311.00        1.35        0.65

In a random 5,000, the screens split into two families. Speed and long string share 83 flags, and Mahalanobis and \(Z_h\) 129 of 250 (rank correlation 0.84); speed and \(Z_h\) share 2, long string and even–odd none. The 311 respondents who gave one option on every screen have a median \(Z_h\) of +1.35, against 0.65 overall: they overfit.

Does a mixed partial credit model find response styles? The fits take twelve minutes, so they were run when the page was built (code).

Code 
# Mixed partial credit models (1-3 classes) on two samples of 2,000 PSS
# respondents, from code/mixture-models-pcm.R (about 20 minutes). ends = share of
# responses in the two end categories, middle = in the middle one; mean_sum is the
# keyed sum (0-56); median_rt in seconds per item.
pcm <- read.csv("data/mixture-models-pcm.csv")
fits <- unique(pcm[, c("sample", "K", "logLik", "npar", "BIC")])
fits[, c("logLik", "BIC")] <- round(fits[, c("logLik", "BIC")])
fits
sample K logLik npar BIC
1 slower 1 -34630 57 69693
2 slower 2 -32021 115 64916
4 slower 3 -31177 173 63669
7 everyone 1 -36855 57 74143
8 everyone 2 -31616 115 64106
10 everyone 3 -30245 173 61805
Code 
# Each class: its share, how it uses the five categories (0 = least stress), and more.
cbind(pcm[pcm$K > 1, c("sample", "K", "class")],
      round(pcm[pcm$K > 1, c("share", "cat0", "cat1", "cat2", "cat3", "cat4", "ends",
                             "mean_sum", "alpha", "median_rt")], 2))
sample K class share cat0 cat1 cat2 cat3 cat4 ends mean_sum alpha median_rt
2 slower 2 1 0.25 0.59 0.18 0.05 0.05 0.11 0.71 12.70 0.73 2.98
3 slower 2 2 0.75 0.13 0.45 0.24 0.15 0.03 0.16 20.91 0.86 4.02
4 slower 3 1 0.22 0.62 0.16 0.05 0.05 0.12 0.75 12.58 0.73 2.84
5 slower 3 2 0.34 0.08 0.32 0.33 0.23 0.04 0.12 25.48 0.77 3.98
6 slower 3 3 0.45 0.20 0.53 0.17 0.08 0.03 0.23 16.88 0.85 3.99
8 everyone 2 1 0.25 0.57 0.07 0.03 0.04 0.29 0.86 19.65 0.79 1.06
9 everyone 2 2 0.75 0.18 0.42 0.22 0.15 0.04 0.21 20.23 0.84 3.84
10 everyone 3 1 0.60 0.11 0.44 0.25 0.16 0.03 0.14 21.85 0.83 3.86
11 everyone 3 2 0.14 0.47 0.04 0.01 0.05 0.44 0.91 27.24 0.23 0.54
12 everyone 3 3 0.26 0.56 0.24 0.07 0.05 0.09 0.64 12.25 0.76 3.13

Among 2,000 slower respondents, BIC falls from 69,693 (one class) to 64,916 (two) and 63,669 (three). With two classes, a quarter of respondents put 71% of their responses in the end categories, against 16% for the rest. An extreme style? That class’s mean keyed sum is also 12.7 against 20.9, with 59% of its responses in the lowest-stress category: style and level arrive together, as in the widget. Among 2,000 drawn from everyone, the three-class fit gives the fast respondents their own class: 14% of the sample, 0.54 seconds per item, a mean keyed sum of 27.2 (the midpoint is 28) and an alpha of 0.23. A model built for styles found the content-free class too.

Fast zero, slow zero

Our last table is test_taking_much_2025_mr: 20 constructed-response matrix reasoning items given on Prolific, with times (Much et al., 2025), and the case Person fit couldn’t see. Its job here is to show what times add.

Code 
# Much et al. (2025): 20 constructed-response matrix items (1 = correct), with
# times in seconds. Items Y_MRm01 and Y_MRt01-06 are practice items, dropped.
much <- read.csv("https://redivis.com/api/v1/tables/datapages.item_response_warehouse_2:v25_0.test_taking_much_2025_mr/rows?format=csv")
much <- much[grepl("^Y_MR[0-9]", much$item), ]
M  <- as.matrix(long2wide(much))
ML <- log(tapply(much$rt, list(much$id, much$item), function(x) x[1]))[rownames(M), colnames(M)]
info <- much[!duplicated(much$id), ]; info <- info[match(rownames(M), info$id), ]
score <- rowSums(M, na.rm = TRUE)
c(respondents = nrow(M), score_zero = sum(score == 0))
respondents  score_zero 
       1243         231
Code 
mean_logt <- rowMeans(ML, na.rm = TRUE)   # each respondent's mean log time per item

# Two classes: engaged respondents follow the Rasch model (theta ~ normal(0, s^2));
# disengaged respondents answer every item correctly with one common probability g.
# With use_time = TRUE each class also has its own normal distribution of mean log
# time. EM on a grid of abilities, extending fit_mixture() from the lesson on
# guessing and priors. A few seconds.
fit_rt_mixture <- function(X, logt, use_time, Q = 41, maxit = 2000, tol = 1e-7) {
  pl <- function(x) pmin(pmax(plogis(x), 1e-10), 1 - 1e-10)
  obs <- !is.na(X); X0 <- ifelse(obs, X, 0)
  z <- seq(-5, 5, length.out = Q); wz <- dnorm(z) / sum(dnorm(z))
  b <- -qlogis(pmin(pmax(colMeans(X, na.rm = TRUE), 0.01), 0.99)); s <- 1; pi <- 0.8; g <- 0.05
  mu <- c(mean(logt), mean(logt) - 0.5); sdv <- c(sd(logt), sd(logt)); old <- -Inf
  for (it in 1:maxit) {
    P <- pl(outer(s * z, b, "-"))
    a <- X0 %*% t(log(P)) + (obs * (1 - X0)) %*% t(log(1 - P)) + rep(log(wz), each = nrow(X))
    mx <- apply(a, 1, max); llR <- mx + log(rowSums(exp(a - mx))); llE <- llR
    llD <- rowSums(X0) * log(g) + rowSums(obs * (1 - X0)) * log(1 - g)
    if (use_time) {
      llE <- llE + dnorm(logt, mu[1], sdv[1], log = TRUE)
      llD <- llD + dnorm(logt, mu[2], sdv[2], log = TRUE)
    }
    both <- cbind(log(pi) + llE, log(1 - pi) + llD)
    mm <- apply(both, 1, max); lli <- mm + log(rowSums(exp(both - mm))); ll <- sum(lli)
    pE <- exp(both[, 1] - lli); pD <- 1 - pE
    Wt <- exp(a - llR) * pE                     # posterior over the grid, engaged class
    nq <- t(Wt) %*% obs; r <- t(Wt) %*% X0
    pi <- mean(pE); g <- sum(pD * rowSums(X0)) / sum(pD * rowSums(obs))
    for (k in 1:5) {
      P <- pl(outer(s * z, b, "-"))
      b <- b + pmax(pmin(colSums(nq * P - r) / colSums(nq * P * (1 - P)), 1), -1)
    }
    s <- exp(optimize(function(ls) { P <- pl(outer(exp(ls) * z, b, "-"))
      -sum(r * log(P) + (nq - r) * log(1 - P)) }, c(-2, 1.5))$minimum)
    if (use_time) {
      mu  <- c(sum(pE * logt) / sum(pE), sum(pD * logt) / sum(pD))
      sdv <- sqrt(c(sum(pE * (logt - mu[1])^2) / sum(pE), sum(pD * (logt - mu[2])^2) / sum(pD)))
    }
    if (abs(ll - old) < tol) break
    old <- ll
  }
  list(pi = pi, g = g, p_disengaged = pD, seconds = exp(mu), loglik = ll)
}
resp_only <- fit_rt_mixture(M, mean_logt, use_time = FALSE)
with_time <- fit_rt_mixture(M, mean_logt, use_time = TRUE)
d0 <- resp_only$p_disengaged > 0.5; d1 <- with_time$p_disengaged > 0.5
round(rbind(responses_only = c(pi = resp_only$pi, g = resp_only$g, n_disengaged = sum(d0)),
            with_times = c(pi = with_time$pi, g = with_time$g, n_disengaged = sum(d1))), 3)
                  pi     g n_disengaged
responses_only 0.905 0.000          226
with_times     0.870 0.005          164
Code 
table(score_zero = score == 0, disengaged_with_times = d1)
          disengaged_with_times
score_zero FALSE TRUE
     FALSE   998   14
     TRUE     81  150
Code 
# Median seconds per item in each class (with times):
round(tapply(exp(mean_logt), ifelse(d1, "disengaged", "engaged"), median))
disengaged    engaged 
        17         37
Code 
# Outside evidence: self-reported lack of understanding, and attention checks passed.
no_understanding <- grepl("lack of understanding", info$cov_disruptions)
round(tapply(no_understanding, ifelse(d1, "disengaged", "engaged"), mean), 2)
disengaged    engaged 
      0.25       0.07
Code 
round(tapply(d1, info$cov_ac, mean), 2)
   1    2    3 
0.33 0.15 0.13

Of 1,243 respondents, 231 got every item wrong. From responses alone, the second class is simply the zero scorers (226, with a success rate of 0.000). Adding times moves 81 slow zero scorers to the engaged class and 14 fast non-zero scorers out: 164 respondents are disengaged (\(\hat\pi\) = 0.87), at a median of 17 seconds per item against 37. A quarter of that class reported not understanding the task, partly or at all, against 7% of the rest; the attention checks barely separate them (13% of those passing all three are in the class, 15% of those passing two).

So what does a class license? The assumptions are on the table: the classes are kinds of respondent, not a continuum the model gets wrong; the cuts and \(K\) were fixed before the substantive analysis; outside evidence agrees. The two kinds of evidence answer different questions. Held-out responses say whether a class predicts: on the Kay data two classes gained 0.17 over the Rasch model, but the 2PL gained 0.16 on its own and the mixture only 0.0088 more, so comparing the mixture with its rivals, not only with one class, is what tells you how much the class adds. The platform, the nonsense items and the times say what the class is. Where a score feeds a decision about a person, as in a campus screening, a high posterior is a probability about how someone answered, and a class is a hypothesis about that, not a verdict on who they are.

Problems

  1. Derivation. Write the posterior class probability under a two-class mixture Rasch model, and show that beyond the sum it depends on the pattern only through \(\sum_i x_i(b_{i2} - b_{i1})\). Then show that on a balanced \(2m\)-item 1–5 scale a same-option respondent always scores the midpoint. What happens on an all-forward scale, and when every raw response is raised by an acquiescence \(\delta\) (ignoring the ceiling)?
  2. Real data with a twist: strategies. balance_mokken holds 25 balance-scale tasks answered by 484 children (van Maanen, Been & Sijtsma, 1989; read it from the CSV link on its landing page). Fit mixture Rasch models with two and three classes. Do the classes’ profiles cross, and do they match the rules Siegler (1976) proposed (Jansen & van der Maas, 1997)? What would make a class a strategy rather than a level?
  3. Judgment. In depression_anxiety_stress, a voluntary online DASS (Lovibond & Lovibond, 1995), some respondents answered “applied to me very much” to all 42 items. Is that a straight line, or severe distress? Use their times, the fake words in the checklist that followed, and the design (items were shown in a random order). What can’t the data tell, and what would you do in a campus screening where a score leads to a follow-up call?
  4. Design. Plan a 20-item survey on which both screens and a mixture can find inattentive respondents: keying, antonym pairs, instructed items, nonsense items, timing. Say what each buys, and set the cuts before collecting data. Then a colleague drops everyone in the Kay content-free class before correlating extraversion with political orientation (cov_poli_cont). What changes, and what should happen to respondents with posteriors near 0.5 (Credé, 2010)?
  5. Real data: latent DIF. Which Kay items function differently between the two classes, and how closely does the class line up with the platform (Cohen & Bolt, 2005; if you’ve done Differential item functioning, compare)? When is a class an observed group under another name? Then refit on the three neuroticism pairs (mood, upst, rlxd): is the content-free class a property of the respondent or of the scale?
  6. Challenge (open). Fit a mixed graded or partial credit model to the full seven-point Kay ratings and look for style classes beside the content-free one. Does the gain survive against a skewed continuum, simulated to match? I don’t know of a general test that tells a real class from a non-normal continuum.

Stuck, or curious about something this lesson doesn’t cover? Ask Claude about this lesson opens a new conversation that already knows what the lesson is about; add your question at the end. Claude shows a caution notice on any prompt that arrives through a link; that’s expected.

Going further

  • Rost, J. (1990). Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement, 14(3), 271–282. https://doi.org/10.1177/014662169001400305. The mixture Rasch model, and its conditional form.
  • Rost, J. (1991). A logistic mixture distribution model for polychotomous item responses. British Journal of Mathematical and Statistical Psychology, 44(1), 75–92. https://doi.org/10.1111/j.2044-8317.1991.tb00951.x. The mixed partial credit model.
  • Mislevy, R. J., & Verhelst, N. (1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika, 55(2), 195–215. https://doi.org/10.1007/bf02295283. Classes as strategies.
  • von Davier, M., & Rost, J. (2016). Logistic mixture-distribution response models. In W. J. van der Linden (Ed.), Handbook of item response theory: Vol. 1. Models. Chapman and Hall/CRC. https://doi.org/10.1201/9781315374512. The handbook chapter this lesson follows.
  • Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085. The screens, compared on one survey.
  • Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006. A practical review of the screens.
  • Schroeders, U., Schmidt, C., & Gnambs, T. (2022). Detecting careless responding in survey data using stochastic gradient boosting. Educational and Psychological Measurement, 82(1), 29–56. https://doi.org/10.1177/00131644211004708. Combining screens into one classifier.
  • Ulitzsch, E., Yildirim-Erbasli, S. N., Gorgun, G., & Bulut, O. (2022). An explanatory mixture IRT model for careless and insufficient effort responding in self-report measures. British Journal of Mathematical and Statistical Psychology, 75(3), 668–698. https://doi.org/10.1111/bmsp.12272. A content-free class modelled with covariates.
  • Ulitzsch, E., Pohl, S., Khorramdel, L., Kroehne, U., & von Davier, M. (2022). A response-time-based latent response mixture model for identifying and modeling careless and insufficient effort responding in survey data. Psychometrika, 87(2), 593–619. https://doi.org/10.1007/s11336-021-09817-7. Responses and item-level times in one mixture.
  • Bauer, D. J., & Curran, P. J. (2003). Distributional assumptions of growth mixture models: Implications for overextraction of latent trajectory classes. Psychological Methods, 8(3), 338–363. https://doi.org/10.1037/1082-989X.8.3.338. Classes from a non-normal continuum.
  • Frick, H., Strobl, C., Leisch, F., & Zeileis, A. (2012). Flexible Rasch mixture models with package psychomix. Journal of Statistical Software, 48(7). https://doi.org/10.18637/jss.v048.i07. Another R route to mixture Rasch models (not tried here).
  • Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6). https://doi.org/10.18637/jss.v048.i06. The package, including multipleGroup(..., dentype = "mixture").

Data sources

Data from the Item Response Warehouse (each table name links to its IRW page); references from IRW biblio.

For instructors

  • Source: new; there is no EDUC 252 class on mixtures. The spine is von Davier and Rost’s handbook chapter. The careless-responding lesson once planned was folded in here (Ben, 09-28).
  • Timing: the core ideas and widgets take about 60 minutes; the Kay analysis works as a 30-minute lab. Locally, the real-data file runs in about five minutes; the held-out comparison (code) and the mixed partial credit fits (code) take about 15 and 12 minutes, so start them first.
  • Sanity check: before drafting, the safety valve of Guessing and priors was refitted on roar_lexical with multipleGroup(), one class fixed at chance through START and FIXED; it reproduced \(\hat\pi\) = 0.87 and the 16 respondents in the guessing class. The same syntax fixes a class in any mirt mixture.
  • Held back: latent DIF (problem 5); strategies on the balance scale (problem 2); the DASS straight lines (problem 3); the joint model of responses and item-level times; factor mixture models (Lubke & Muthén, 2005); faking on high-stakes surveys.
  • Solutions: held back in solutions/mixture-models.qmd (not published).