AI and psychometrics

Module: Beyond the standard models
Tranche: extension (beyond a first course)
Before this: From the 1PL to the 4PL · What is an item? Explanatory item response models
Builds toward: none
Picks up threads: Items are a sample from a population of possible items, describable by their features (from What is an item? Explanatory item response models); Item features predict difficulty, but not all of it (from What is an item? Explanatory item response models)
IRW tables: himmelstein-impossible_question-2025, gpt4mcq_young_2025, genpsych_russell_2024_gpt4o
From EDUC 252: slides c10; code c10/model.R
Status: draft
Authorship: Written largely by Claude (Anthropic), from Ben Domingue’s EDUC 252 materials and under Ben’s direction. Ben reviews each lesson before it is marked done.
Comments: open an issue on GitHub about this lesson.

What this is for

Language models now score essays, write items, predict item difficulty, and answer questionnaires in place of people. Each is a claim about measurement that the tools of this course can check. The lesson starts from the question What is an item? handed on: that lesson explained difficulty with features that people coded, so can an item’s text alone predict it? The answer depends on the yardstick. Judged by how many respondents a difficulty prediction is worth, the best text features for the general-knowledge items below are worth about three.

Goals

By the end of this lesson you should be able to:

  • compute exact agreement, Cohen’s kappa and quadratic weighted kappa from two raters’ scores, and say what an automated scorer’s agreement with human raters does and doesn’t show;
  • add text features to the LLTM and report how much of the variation in difficulty they explain;
  • convert the error of a difficulty prediction into the number of respondents it is worth;
  • say why more simulated respondents don’t fix a language model’s own view of the items;
  • check machine-written items as you would any others: difficulty, distractors, keying, and whether each item goes with the construct it was written for.

Core ideas

Two intersections

AI and psychometrics meet in two directions. One uses AI to measure people. The other uses psychometrics to measure AI: a benchmark is a set of items and a model is a respondent, and Martínez-Plumed, Prudêncio, Martínez-Usó and Hernández-Orallo (2019) fit item response models to machine-learning classifiers, one item per test instance. The IRW holds no table of models answering questions yet, so that direction gets problem 6; the rest of the lesson is about the first.

Automated scoring is an agreement problem

An essay-scoring engine is trained to reproduce the scores human raters give; e-rater is a long-running example (Attali & Burstein, 2006). How do we know it works? The first evidence is agreement with human raters, read from the table that crosses the engine’s scores with a human’s. Three statistics summarize that table:

  • exact agreement, the share of responses on the diagonal;
  • Cohen’s kappa, exact agreement corrected for the agreement two raters would reach by chance with the same margins, \(\kappa = (p_o - p_e)/(1 - p_e)\);
  • quadratic weighted kappa (QWK; Cohen, 1968), which charges each disagreement by its squared distance: with weights \(w_{ik} = (i - k)^2\), observed counts \(O_{ik}\) and the counts \(E_{ik}\) expected from the margins, \[\text{QWK} = 1 - \frac{\sum_{i,k} w_{ik} O_{ik}}{\sum_{i,k} w_{ik} E_{ik}}.\]

Agreement with what? The baseline is the agreement of two human raters scoring the same responses. Williamson, Xi and Breyer (2012) make human–machine agreement one strand of the evidence for an engine, beside its fit to the purpose, its relations with independent measures, its generalizability across tasks and forms, and its consequences for subgroups.

Try it: agreement by hand

Five hundred essays have a true quality. Two human raters score it on a scale from 1 to 4 with the same noise; an engine adds noise of its own, may shift, and may compress its scores toward the middle.

As the widget loads, the engine agrees with human 1 about as well as human 2 does (QWK 0.70 against 0.65; the gap is sampling noise). Shift it up half a point: exact agreement falls from 0.54 to 0.47, QWK only to 0.65. Compress its scale to 0.6 and QWK drops to 0.55. Then cut its noise to 0.3 with the scale at 0.7: it matches the human baseline (0.66) while still hugging the middle. The same QWK can come from quite different engines.

QWK depends on the margins of the table and the length of the scale (Lewis & Casabianca, 2026). The proportional reduction in mean squared error (PRMSE) judges an engine against the human raters’ true scores, allowing for their unreliability, and can reach a different conclusion from QWK on the same data (McCaffrey, Casabianca & Johnson, 2025). And agreement says nothing about what the engine responds to: in 240,000 admission essays, content and style correlated more strongly with household income than SAT scores did (Alvero et al., 2021). An engine could agree with human raters and carry that signal too; Williamson et al.’s subgroup evidence is how you would find out.

If you’ve done Rater models: an engine is one more rater, with a severity of its own. Casabianca, Donoghue, Shin, Chao and Choi (2023) found that two human ratings connected raters better than one human rating plus an engine’s, and that engine scores alone did not connect raters adequately.

Assumptions: two raters score the same \(n\) responses on an integer scale; \(p_{ik}\) is the observed share with scores \((i, k)\); the expected shares are the products of the margins, \(p_{i\cdot}\,p_{\cdot k}\); weights are \((i - k)^2\); means, variances and the covariance are computed with divisor \(n\).

The numerator of the fraction in QWK is the mean squared difference between the two scores, \(\sum_{i,k} (i-k)^2 p_{ik} = E[(X - Y)^2]\). The denominator is the same expectation for two independent raters with the same margins. Expanding the square, \[E[(X-Y)^2] = \sigma_X^2 + \sigma_Y^2 - 2\sigma_{XY} + (\mu_X - \mu_Y)^2,\] and under independence the covariance term is 0. So \[\text{QWK} = 1 - \frac{\sigma_X^2 + \sigma_Y^2 - 2\sigma_{XY} + (\mu_X - \mu_Y)^2}{\sigma_X^2 + \sigma_Y^2 + (\mu_X - \mu_Y)^2} = \frac{2\sigma_{XY}}{\sigma_X^2 + \sigma_Y^2 + (\mu_X - \mu_Y)^2}.\] This is exact, and it is Lin’s (1989) concordance correlation coefficient. Three things follow. With equal means and variances, QWK is the Pearson correlation. A difference in means lowers it (the widget’s shift), and so does a difference in spread even when the correlation is 1 (the widget’s compression). And Fleiss and Cohen (1973) showed that weighted kappa with these weights is, apart from terms of order \(1/n\), the intraclass correlation that counts rater differences as error: the absolute-agreement coefficient of Many sources of error, if you’ve done it.

Can the text of an item predict its difficulty?

Recall from What is an item?: the LLTM writes each item’s easiness as a weighted sum of its features, and an item residual with SD \(\sigma_\varepsilon\) gives a size to what the features leave out. For the verbal aggression items, features coded from the design explained 89% of the variance in easiness, and the residual SD was 0.34.

Where do the features come from? Some are counts from the text, such as word count and sentence length, the stuff of readability measures like the Lexile framework (Stenner, Burdick, Sanford & Burdick, 2006). Some are coded by experts from a model of how items are solved: for paragraph-comprehension items from the GRE, variables describing the decision among options accounted for difficulty and those describing the text did not (Gorin & Embretson, 2006). And some are representations of the text from a language model (Benedetto et al., 2023, survey the methods). Kapoor, Truong, Haber, Ruiz-Primo and Domingue (2025) predicted the difficulty of state reading items from annotated linguistic, test and context features: a correlation of 0.77 with the true difficulties, and a root mean squared error of 0.59 against 0.92 for predicting the mean. Embeddings from language models improved the prediction only marginally, and either kind of feature alone did about as well.

Is a correlation of 0.77 good? Two yardsticks assume different uses. Accuracy (a correlation, an RMSE) assumes the prediction is the product, as when sorting a bank. Respondents assumes the point is to calibrate items with fewer people. A calibration on \(n\) respondents estimates \(b\) with a standard error of about \(1/\sqrt{n\,E[p(1-p)]}\), since each respondent adds \(p(1-p)\) to the information about the item’s location (the curvature of The likelihood and logistic regression). A prediction whose errors have SD \(\sigma\) is therefore as precise as a calibration on \[n^* = \frac{1}{\sigma^2\,E[p(1-p)]}\] respondents.

This lesson uses the second yardstick, because the use it has in mind is skipping or shrinking the field test, and for that use the question is how many respondents a prediction is worth; accuracy statistics don’t say. Where the prediction itself is the product, the first yardstick answers the question asked.

Try it: what a prediction is worth

With Kapoor et al.’s correlation and spread of difficulties, the prediction is worth about 14 respondents, if their difficulties are logits on a scale where \(\theta\) has SD 1 (an assumption here). A correlation of 0.9 is worth 30; 0.95 is worth 59; matching a 200-person field test needs about 0.986. Liu, Bhandari and Pardos (2025) report their result on this yardstick: the item parameters from their language-model respondents were about as good as those from 50 undergraduates.

If you’ve done Model fit and out-of-sample prediction: the IMV put a predictive gain on a scale that compares across datasets. Respondents do the same job here, in the unit a test developer pays for.

Language models as simulated respondents

Argyle et al. (2023) conditioned GPT-3 on the backgrounds of real survey participants and found that it reproduced the response patterns of many subgroups. For items, Liu et al. (2025) had six models answer college algebra items: each model’s abilities were narrowly spread, an ensemble of models came closer to students, and the item parameters correlated above 0.8 with those from students for GPT-3.5. Zhou, Liu, Tang and Wilson (2026), a preprint, address that narrow spread by telling the model which student to be: each prompt describes a profile of fraction-subtraction skills from a cognitive diagnosis model. With profiles, the models’ scores spread more like students’, and the item difficulties from their answers came closer to the students’ (in the strongest case, a rank correlation of about 0.9 against 0.24 without profiles; see Latent classes and cognitive diagnosis). Scarlatos, Fernandez, Ormerod, Lottridge and Lan (2025) train a model so that simulated students at an instructed ability answer as an item response model says they should, and use the simulated answers to predict the difficulty of open-ended items.

Does simulation beat the simpler route of the last section? A simulator is a difficulty predictor, so it answers to the same yardstick, and it has to get each item right: a model may find an item easy or hard for reasons of its own.

Try it: simulated respondents

Thirty items with known difficulties, answered by a model playing simulated students.

At the defaults the correlation is 0.91 and the simulation is worth about 30 respondents. Raise the simulated respondents to 2,000 and the correlation stays at 0.91. Set the bias to 0 instead and it rises to 0.98, worth about 144 respondents. More simulated respondents remove sampling noise, not the model’s own view of the items. (The widget rescales implied difficulties to the truth’s spread, so the narrow spread matters little here.)

Automatic item generation, and checking what it makes

Items have been generated by rule for decades. A cognitive design system builds in the features that make an item hard and calibrates the design principles rather than each item (Embretson, 1999); item models are templates with slots filled by rule (Gierl & Haladyna, 2012; for medical multiple-choice items, Gierl, Lai & Turner, 2012). Language models write items without a template. AI-GENIE generates a pool with a language model and prunes it with network psychometrics before anyone answers (Russell-Lasalandra, Christensen & Golino, 2024).

Whoever writes the items, the questions are the same: does each measure its construct, how hard is it, do the wrong options draw anyone? Distractors that draw almost nobody are common in human-written tests too: Haladyna and Downing (1993) found that items seldom had more than three options that performed acceptably, and a meta-analysis of 80 years of studies supports three options (Rodriguez, 2005).

Try it: distractors that work, and dead ones

A four-option item. The key’s pull rises with \(\theta\); each distractor has a fixed pull of its own.

At the defaults distractor A draws 38% of the group and B and C draw 3% each: two dead distractors. Respondents far below the item choose A, so the key’s curve falls below chance (dashed). Make the key easier (2.5) and even A falls to 13%. On an easy item, dead distractors alone don’t show that they were badly written.

Recall from From the 1PL to the 4PL: with four options, chance is 0.25, yet the 3PL’s lower asymptotes for the RMET were mostly near 0, because almost nobody sat near chance. A working distractor is one more reason the floor needn’t be at chance.

Simulate

The code simulates 200 items whose difficulties follow two text features plus a residual with SD sd_resid, fits the feature model on 150, predicts the other 50, converts the error to respondents, and compares a field test of the 50 with n_field respondents. Base R; a second.

With the seed as written, the prediction correlates 0.68 with the new difficulties (RMSE 0.57), worth about 16 respondents; a 25-person field test does better (RMSE 0.43). Set n_field <- 16 and the field test’s RMSE is 0.58, matching the prediction. Try sd_resid <- 0.2 (worth about 142) and sd_resid <- 1 (about 6).

Download this code to run it in your own R session.

With real data

Our main example is himmelstein-impossible_question-2025: its job is the deep dive in miniature, testing whether item text predicts difficulty. The two contrasts are items written by language models: gpt4mcq_young_2025, multiple-choice items written by GPT-4, and genpsych_russell_2024_gpt4o, personality items written by GPT-4o. Both are openly licensed, so their items are shown in full (citing the authors and naming the model). Download the code; it needs lme4, mirt and haven.

Impossible questions: does the text predict difficulty?

The Forecasting Proficiency Test (Himmelstein et al., 2025; materials and data at forecastingresearch/fpt) includes an impossible-question task adapted from Bennett and Steyvers (2022). Each question offers two answers and an opt-out. Most have a right answer; some ask about things that don’t exist, and the right response is to opt out. Two examples from the study’s data file are “Not all metals are attracted to magnets” (true or false) and “What’s the habitat of a Rojixon bird?” (deserts or tropical rainforests). Keying: 1 is the keyed answer, or an opt-out on an impossible question. A gentle note from the IRW’s item-text notes: the instructions said opt-outs weren’t scored, but on answerable questions they count as wrong.

Code 
suppressPackageStartupMessages(library(lme4))
# Each IRW table has a landing page (itemresponsewarehouse.org/tables/<name>/) with a
# CSV download. This link is pinned to one version of the data.
iq_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse_2:v24_0.himmelstein-impossible_question-2025/rows?format=csv"
iq <- read.csv(iq_url)
c(respondents = length(unique(iq$id)), items = length(unique(iq$item)),
  max_per_person_item = max(table(iq$id, iq$item)))
        respondents               items max_per_person_item 
               1194                  90                   1
Code 
# Two waves: each respondent took 30 items in wave 3 and 30 in wave 5.
table(wave = iq$wave)
wave
    3     5 
35179 34924
Code 
summary(as.vector(table(iq$id)))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  28.00   59.00   60.00   58.71   60.00   60.00
Code 
# The questions, from the study's own data file (pinned to one commit of the FPT
# repository): the IRW item code is the file's `id`. `aig_version` says which form an
# item was on: the anchor form, which every respondent took, or one of two forms
# built for automatic item generation (A and B). Impossible questions have ids IQ_*.
src_url <- paste0("https://raw.githubusercontent.com/forecastingresearch/fpt/",
                  "fdbe605700cc172c03abe009bbd894a20f5cb69b/",
                  "data_cognitive_tasks/task_datasets/data_impossible_question.csv")
src <- read.csv(src_url)
items <- unique(src[, c("id", "question_text", "answer_1", "answer_2", "question_type", "aig_version")])
names(items)[1] <- "item"
table(type = items$question_type, form = items$aig_version)
    form
type aig_A aig_B anchor
  GK    24    24     24
  IQ     6     6      6
Code 
# Which form did each respondent take in which wave? The anchor form in one wave and
# one generated form in the other, so pooling the waves counts each person once per item.
iq$form <- items$aig_version[match(iq$item, items$item)]
table(wave = iq$wave, form = iq$form)
    form
wave aig_A aig_B anchor
   3  8794  8151  18234
   5  9309  8995  16620
Code 
# Two text features, as in the IRW vignette on item language and difficulty: word
# count, and mean word length in characters with punctuation removed.
text_features <- function(txt) {
  words <- strsplit(trimws(gsub("[[:punct:]]", " ", txt)), "\\s+")
  words <- lapply(words, function(w) w[nchar(w) > 0])
  data.frame(word_count = lengths(strsplit(trimws(txt), "\\s+")),
             mean_word_length = vapply(words, function(w) if (length(w)) mean(nchar(w)) else NA_real_, 0))
}
items <- cbind(items, text_features(items$question_text))
items$impossible <- as.integer(items$question_type == "IQ")
# Proportion correct among those who answered, and its logit. (An opt-out is scored
# correct on an impossible question and wrong on an answerable one.)
items$p <- as.vector(tapply(iq$resp, iq$item, mean)[items$item])
items$n <- as.vector(table(iq$item)[items$item])
items$logit_p <- qlogis((items$p * items$n + 0.5) / (items$n + 1))
head(items[order(items$p), c("item", "question_text", "p")], 4)
item question_text p
4 IQ_6 Resistance Configuration Theory is a psychological theory that explains: 0.1922078
196 IQ_18 The Diflective Assumption Theory helps us understand: 0.2447552
24 IQ_2 How many sides does a Detseroid have? 0.2724167
205 IQ_16 What’s the side effect of Grilpax’s disease? 0.2732049

Each respondent took the 30-item anchor form in one wave and one of two generated forms in the other, and no one answered an item twice, so we pool the waves. The hardest items are impossible questions, answered correctly (by opting out) by as few as 19%.

Code 
# Word count against logit proportion correct: all 90 items, the 72 answerable ones,
# and mean word length for comparison.
ans <- items$impossible == 0
round(c(words_all = cor(items$word_count, items$logit_p),
        words_answerable = cor(items$word_count[ans], items$logit_p[ans]),
        word_length_all = cor(items$mean_word_length, items$logit_p)), 2)
       words_all words_answerable  word_length_all 
           -0.11            -0.26             0.15
Code 
# What the text says: the impossible flag.
aggregate(cbind(p, word_count) ~ question_type, data = items, FUN = function(v) round(mean(v), 2))
question_type p word_count
GK 0.70 11.36
IQ 0.34 7.50
Code 
round(c(r2_impossible_flag = summary(lm(logit_p ~ impossible, data = items))$r.squared,
        r2_word_count = summary(lm(logit_p ~ word_count, data = items))$r.squared), 2)
r2_impossible_flag      r2_word_count 
              0.23               0.01

Longer answerable questions are somewhat harder (r = −0.26). Across all 90 the correlation is −0.11, because the impossible questions are both shorter (7.5 words against 11.4) and much harder (0.34 correct against 0.70), which pulls the other way. What the text says matters more than its length: the flag for an impossible question explains 23% of the variance in logit proportion correct, word count 1%. Now the LLTM with an item residual, the text features as item predictors:

Code 
# The LLTM with an item residual (from "What is an item?"), with text features as the
# item predictors: logit P(correct) = theta_j + beta_0 + features + epsilon_i. The item
# residual's variance, with and without the features, says how much of the variation
# in difficulty they explain. nAGQ = 0 keeps each fit to seconds.
d <- merge(iq, items[, c("item", "word_count", "impossible")], by = "item")
d$words_z <- (d$word_count - mean(items$word_count)) / sd(items$word_count)
m0 <- glmer(resp ~ 1 + (1 | id) + (1 | item), family = binomial, data = d, nAGQ = 0)
m1 <- glmer(resp ~ words_z + (1 | id) + (1 | item), family = binomial, data = d, nAGQ = 0)
m2 <- glmer(resp ~ words_z + impossible + (1 | id) + (1 | item), family = binomial, data = d, nAGQ = 0)
sds <- sapply(list(none = m0, word_count = m1, word_count_and_flag = m2), function(m) {
  v <- as.data.frame(VarCorr(m)); setNames(v$sdcor, v$grp)[c("item", "id")] })
round(rbind(sds, share_of_item_variance_explained = 1 - sds["item", ]^2 / sds["item", "none"]^2), 3)
                                  none word_count word_count_and_flag
item                             1.709      1.698               1.420
id                               0.466      0.466               0.466
share_of_item_variance_explained 0.000      0.014               0.310
Code 
round(fixef(m2), 2)
(Intercept)     words_z  impossible 
       1.38       -0.40       -2.28

Items differ far more than respondents (SD 1.71 against 0.47 for \(\theta\)). Word count explains 1.4% of the item variance; adding the impossible flag, 31%, leaving a residual SD of 1.42. What is that worth?

Code 
# What is each prediction worth, in respondents? One respondent carries E[p(1 - p)]
# units of information about an item's location (Rasch model, theta ~ N(0, sd_theta));
# a prediction with error SD sigma is as good as a calibration on
# n* = 1 / (sigma^2 * E[p(1 - p)]) respondents. E[p(1 - p)] is averaged over items at
# the locations each model predicts.
epq <- function(loc, sd_theta) integrate(function(t)
  plogis(loc + t) * (1 - plogis(loc + t)) * dnorm(t, 0, sd_theta), -Inf, Inf)$value
worth <- function(m) {
  v <- as.data.frame(VarCorr(m)); sigma <- v$sdcor[v$grp == "item"]; sd_theta <- v$sdcor[v$grp == "id"]
  loc <- unique(model.matrix(m) %*% fixef(m))
  info <- mean(sapply(loc, epq, sd_theta = sd_theta))
  c(sigma = sigma, info_per_respondent = info, respondents_worth = 1 / (sigma^2 * info))
}
round(sapply(list(none = m0, word_count = m1, word_count_and_flag = m2), worth), 2)
                    none word_count word_count_and_flag
sigma               1.71       1.70                1.42
info_per_respondent 0.20       0.21                0.18
respondents_worth   1.73       1.69                2.69

Knowing only that an item belongs to this test is worth about 1.7 respondents: that is the prior. Word count adds nothing; with the impossible flag, the prediction is worth 2.7. On the respondents yardstick, what the text says about these items is worth about one respondent. Questions of the same kind still differ by 1.42 logits for reasons these features don’t see.

The table holds one more check. Twelve of the impossible questions, on the two generated forms, were written by ChatGPT for this study; the six on the anchor form come from the original bank (Himmelstein et al., 2025).

Code 
# Six impossible questions are from the original item bank (the anchor form); the
# twelve on the two generated forms were written by ChatGPT for this study.
iqs <- items[items$impossible == 1, ]
iqs$written_by <- ifelse(iqs$aig_version == "anchor", "original bank", "ChatGPT")
aggregate(cbind(p, word_count) ~ written_by, data = iqs, FUN = function(v) round(mean(v), 2))
written_by p word_count
ChatGPT 0.34 7.58
original bank 0.32 7.33
Code 
iqs[order(iqs$written_by, iqs$p), c("item", "written_by", "question_text", "p")]
item written_by question_text p
196 IQ_18 ChatGPT The Diflective Assumption Theory helps us understand: 0.2447552
205 IQ_16 ChatGPT What’s the side effect of Grilpax’s disease? 0.2732049
190 IQ_15 ChatGPT What’s the habitat of a Rojixon bird? 0.2827225
125 IQ_7 ChatGPT When did the Laikrev Uprising occur? 0.3211921
199 IQ_17 ChatGPT What is the Kraghold method used for? 0.3222997
139 IQ_12 ChatGPT How does Binoxelation Impact Theory explain the growth of crystals? 0.3382838
129 IQ_11 ChatGPT What does the Veklin Procedure involve? 0.3603306
149 IQ_9 ChatGPT What is the speed of a Blargan Horse? 0.3698176
145 IQ_8 ChatGPT What is the nutritional value of a Golgrest Fruit? 0.3904448
185 IQ_13 ChatGPT What are the ingredients of a traditional Zibrune stew? 0.3937282
210 IQ_14 ChatGPT What’s the main purpose of a Tantrigon device? 0.3989547
147 IQ_10 ChatGPT What is the cure for Zolbrod’s syndrome? 0.4347107
4 IQ_6 original bank Resistance Configuration Theory is a psychological theory that explains: 0.1922078
24 IQ_2 original bank How many sides does a Detseroid have? 0.2724167
8 IQ_5 original bank What is the Parlichev method? 0.3466780
19 IQ_4 original bank What is the most prominent symptom of Seradot’s disease? 0.3634812
7 IQ_1 original bank When did the battle of Kavkav take place? 0.3810742
1 IQ_3 original bank What is the population of Synomle? 0.3858603

ChatGPT’s impossible questions are opted out of 34% of the time, the originals 32%, with similar lengths: the generated items behave like the ones they were written to match, the anchor-form check the study used for exchangeability.

GPT-4’s multiple-choice items

Young, Courtney, Kah, Wilkerson and Chen (2025) gave GPT-4 a textbook chapter on love, friendship and social support and asked for 20 multiple-choice items, which 190 undergraduates answered after reading it. The items (CC BY 4.0) come from the OSF deposit. Keying: resp is the option chosen, and option 1 is the key on every item, which the IRW confirmed against the published proportions correct.

Code 
library(haven)
suppressPackageStartupMessages(library(mirt))
mcq_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v60_0.gpt4mcq_young_2025/rows?format=csv"
mcq <- read.csv(mcq_url)
# resp is the option chosen, 1 to 4; option 1 is the key on every item.
wide <- tapply(mcq$resp, list(mcq$id, mcq$item), function(v) v[1])[, paste0("AIQ", 1:20)]
c(respondents = nrow(wide), missing = sum(is.na(wide)))
respondents     missing 
        190           0
Code 
# The items, from the SPSS labels in the authors' OSF deposit (osf.io/zq4eg, file
# version 1): stems in the variable labels, options in the value labels.
sav_file <- tempfile(fileext = ".sav")
download.file("https://osf.io/download/bj3gc/?version=1", sav_file, mode = "wb", quiet = TRUE)
sav <- read_sav(sav_file)
stem <- sapply(colnames(wide), function(v) attr(sav[[v]], "label"))
opts <- lapply(colnames(wide), function(v) { l <- attr(sav[[v]], "labels"); names(l)[order(l)] })
key_len <- sapply(opts, function(o) nchar(o[1])); dis_len <- sapply(opts, function(o) max(nchar(o[-1])))
Code 
key <- (wide == 1) * 1
mcq_items <- data.frame(item = colnames(wide), stem = unname(stem), key = sapply(opts, `[`, 1),
                        p = round(colMeans(key), 3), key_is_longest = key_len > dis_len)
knitr::kable(mcq_items[, c("item", "stem", "key", "p")], row.names = FALSE)
item stem key p
AIQ1 Which factor is NOT listed as influencing attraction between people according to psychological research? Financial status 0.916
AIQ2 According to Durkheim, what is a crucial factor for personal well-being and potentially preventing suicide? Being socially connected 0.889
AIQ3 How does the Internet and social media influence the development of relationships, according to the chapter? By providing a virtual space where people can meet and develop deep connections 0.816
AIQ4 What does Sternberg’s Triangular Theory of Love suggest about the components of love? Love is comprised of intimacy, passion, and commitment 0.926
AIQ5 Which of the following is NOT a conceptual component of social support? Financial assistance 0.911
AIQ6 How does familiarity affect attraction, according to psychological research? Repeated exposure to someone increases the likelihood of attraction 0.937
AIQ7 What role does reciprocity play in the formation of friendships and romantic relationships? We are more likely to like someone if they show that they like us in return 0.932
AIQ8 According to research, how does having close friendships affect individuals during difficult times? Close friendships can buffer individuals from stress and protect their self-esteem 0.932
AIQ9 What does Helen Fisher’s research on brain chemistry in individuals who have just fallen in love suggest? Brain chemistry resembles that of an addict experiencing a high 0.926
AIQ10 How does perceived social support influence well-being? Believing that others will be available in times of need can increase happiness and well-being 0.889
AIQ11 Why are online relationships sometimes considered more intimate than in-person relationships? They offer a safe space for more socially anxious individuals to develop meaningful connections 0.889
AIQ12 How does the matching hypothesis explain attraction? People are attracted to others who are similar to them in various attributes 0.926
AIQ13 What impact does workplace friendship have on employees’ attitudes towards their job? Increases job satisfaction, job involvement, and organizational commitment 0.947
AIQ14 What is a significant benefit of having a diverse social network? Less likely to get the common cold compared to those with less diverse networks 0.679
AIQ15 How does the Internet change the dynamic of forming relationships, according to the chapter? It reduces issues of proximity and allows for communication across distances 0.832
AIQ16 According to Baumeister and Leary, what is a fundamental human need? The need to belong 0.926
AIQ17 What does the mere-exposure effect suggest about attraction? We are more likely to be attracted to someone we are frequently exposed to 0.937
AIQ18 What does research say about the importance of similarity in friendships and romantic relationships? Similarity in values, attitudes, and interests plays a key role in forming strong relationships 0.937
AIQ19 What role does received social support play in an individual’s life, according to the chapter? It can buffer people from stress but may not always be viewed as helpful if it is unwanted 0.500
AIQ20 How does Sternberg’s model differentiate between types of love? By identifying different combinations of intimacy, passion, and commitment present in each type 0.868
Code 
c(median_p = median(mcq_items$p), key_longest = sum(mcq_items$key_is_longest))
   median_p key_longest 
      0.921      12.000

The items are easy for students who had just read the chapter: the median proportion correct is 0.92. The key is the longest option in 12 of the 20 items, a cue that item-writing guidelines advise against (Haladyna, Downing & Rodriguez, 2002).

Code 
# Share of respondents choosing each option; rows 2-4 are the distractors.
shares <- sapply(1:20, function(i) prop.table(table(factor(wide[, i], levels = 1:4))))
colnames(shares) <- colnames(wide)
round(shares[, c("AIQ1", "AIQ14", "AIQ19")], 3)
   AIQ1 AIQ14 AIQ19
1 0.916 0.679 0.500
2 0.016 0.084 0.453
3 0.042 0.163 0.011
4 0.026 0.074 0.037
Code 
dis <- shares[2:4, ]
c(distractors = length(dis), chosen_by_under_5_percent = sum(dis < 0.05), chosen_by_nobody = sum(dis == 0))
              distractors chosen_by_under_5_percent          chosen_by_nobody 
                       60                        48                         2
Code 
opts[[19]]
[1] "It can buffer people from stress but may not always be viewed as helpful if it is unwanted"
[2] "Received social support has universally positive effects on individuals"                   
[3] "It has no significant impact on stress levels or well-being"                               
[4] "Received support is less important than perceived support in coping with stress"

Of the 60 distractors, 48 are chosen by fewer than 5% of students and 2 by nobody, as in the widget’s easy item. The exception is AIQ19, on received social support: 45% chose “Received social support has universally positive effects on individuals”, a distractor that works.

Code 
# Reliability and item parameters. Alpha as in "Classical test theory"; then the 2PL,
# with mirt's intercept d converted to a difficulty, b = -d / a.
alpha <- function(x) { k <- ncol(x); k / (k - 1) * (1 - sum(apply(x, 2, var)) / var(rowSums(x))) }
table(sum_score = rowSums(key))
sum_score
 0  3  4  5  7  8 10 11 12 13 14 15 16 17 18 19 20 
 1  1  2  1  1  3  2  2  1  5  8  6  9 11 33 45 59
Code 
set.seed(55)
fit2 <- mirt(as.data.frame(key), 1, itemtype = "2PL", verbose = FALSE)
cf <- coef(fit2, simplify = TRUE)$items
b <- -cf[, "d"] / cf[, "a1"]
round(c(alpha = alpha(key), median_b = median(b), min_b = min(b), max_b = max(b)), 2)
   alpha median_b    min_b    max_b 
    0.89    -1.64    -2.88     0.00

Alpha is 0.89, yet 59 of the 190 students answered every item correctly. The alpha comes largely from the few who missed many items, and the 2PL difficulties (median −1.64) sit well below the average student. The test separates students who struggled with the chapter from the rest, which suits a check on reading, the use the authors recommend.

GPT-4o’s personality items

Russell-Lasalandra, Christensen and Golino (2024) generated Big Five items with GPT-4o through AI-GENIE and gave the 35 that survived to a nationally representative US sample; items and traits come from their OSF deposit (CC0). Keying: 1 is strongly disagree and 5 strongly agree, and every item is worded toward its trait.

Code 
b5_url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v60_0.genpsych_russell_2024_gpt4o/rows?format=csv"
b5 <- read.csv(b5_url)
# resp runs from 1 (strongly disagree) to 5 (strongly agree). The items, and the trait
# each was written for, from the authors' OSF deposit (osf.io/zcytb, file version 1),
# in the same order as items_1 to items_35.
b5_items <- read.csv("https://osf.io/download/k7e5t/?version=1")
b5_items$item <- paste0("items_", seq_len(nrow(b5_items)))
w5 <- tapply(b5$resp, list(b5$id, b5$item), function(v) v[1])[, b5_items$item]
c(respondents = nrow(w5), complete = sum(complete.cases(w5)))
respondents    complete 
        999         897
Code 
w5 <- w5[complete.cases(w5), ]
table(trait = b5_items$characteristic)
trait
    agreeableness conscientiousness      extraversion       neuroticism 
                7                 8                 5                 8 
         openness 
                7

We keep the 897 respondents who answered every item. Does each item go with its trait? Each item’s correlation with the sum of the other items of each trait:

Code 
# Does each item go with the trait it was written for? Correlate it with the sum of
# the other items of each trait (its own trait's sum leaves it out).
trait <- b5_items$characteristic
traits <- unique(trait)
ir <- t(sapply(seq_len(ncol(w5)), function(i) sapply(traits, function(t) {
  cols <- setdiff(which(trait == t), i); cor(w5[, i], rowSums(w5[, cols, drop = FALSE])) })))
own <- ir[cbind(seq_along(trait), match(trait, traits))]
other <- sapply(seq_along(trait), function(i) max(ir[i, traits != trait[i]]))
c(items_closest_to_own_trait = sum(own > other), items = length(own))
items_closest_to_own_trait                      items 
                        35                         35
Code 
round(c(median_own = median(own), median_best_other = median(other)), 2)
       median_own median_best_other 
             0.60              0.21
Code 
b5_items[which.min(own - other), c("item", "characteristic", "item_content")]
item characteristic item_content
32 items_32 openness I love experimenting with new recipes and cooking techniques.
Code 
round(ir[which.min(own - other), ], 2)
    agreeableness conscientiousness      extraversion       neuroticism 
             0.31              0.13              0.23             -0.07 
         openness 
             0.34

All 35 items correlate most with their own trait (median 0.60, against 0.21 for the closest other trait). The narrowest margin is “I love experimenting with new recipes and cooking techniques”, written for openness: 0.34 with openness, 0.31 with agreeableness.

Code 
# Alpha for each trait's items, and the mean correlation among them.
t(sapply(traits, function(t) { x <- w5[, trait == t]; r <- cor(x)
  round(c(items = ncol(x), alpha = alpha(x), mean_r = mean(r[upper.tri(r)])), 2) }))
                  items alpha mean_r
agreeableness         7  0.77   0.33
conscientiousness     8  0.83   0.38
extraversion          5  0.85   0.54
neuroticism           8  0.93   0.61
openness              7  0.77   0.34
Code 
b5_items$item_content[b5_items$characteristic == "neuroticism"]
[1] "I frequently worry about small things."                     
[2] "I often find myself feeling anxious without a clear reason."
[3] "I often feel overwhelmed by my emotions."                   
[4] "I frequently experience sudden mood changes."               
[5] "I frequently feel sad or down for no specific reason."      
[6] "I find myself getting upset easily over minor issues."      
[7] "I often feel tense or stressed."                            
[8] "I frequently feel nervous about the future."

The eight neuroticism items reach alpha 0.93 with a mean correlation of 0.61, and read almost as rewordings of each other. High homogeneity can mean items that repeat one another rather than a broad construct measured well (Boyle, 1991).

Recall from Classical test theory and reliability: items worded against the construct have to be reverse-keyed. Here no item is: keying is easy, and a respondent’s tendency to agree is mixed into every trait score. If you’ve done From construct map to items, that is the all-forward scale whose alpha rose with acquiescence.

Each set of machine-written items needed the usual checks, and each turned something up. A machine-written item is still an item.

Problems

  1. QWK from a table. Show that QWK \(= 1 - \sum w_{ik}O_{ik}/\sum w_{ik}E_{ik}\) with weights \((i-k)^2\) equals \(2\sigma_{XY}/(\sigma_X^2 + \sigma_Y^2 + (\mu_X - \mu_Y)^2)\). With two categories, show that it equals Cohen’s kappa.
  2. Without the impossible questions. Drop the 18 impossible questions and refit the LLTM with word count and mean word length. What do they explain, and what is the prediction worth? Code a feature of your own from the text (true–false or not, say) and test it.
  3. A 2PL for GPT-4’s items. Plot the 2PL test information for gpt4mcq_young_2025. Where is it informative? Which three items would you ask the writer to make harder, and how?
  4. Judgment. A vendor’s engine agrees with human raters at QWK 0.80 on your essays; two humans agree at 0.72. What else do you need before it replaces the second rater?
  5. Design. Design a study of whether difficulties predicted by a language model could replace a state’s field test of new reading items. What result would convince you?
  6. Challenge (open): models as respondents. A hundred language models answer a thousand benchmark questions. What would “ability” mean for a model, and which assumption is likeliest to fail: local independence (models trained on overlapping data), one dimension, or something else? I don’t know the answer.

Stuck, or curious about something this lesson doesn’t cover? Ask Claude about this lesson opens a new conversation that already knows what the lesson is about; add your question at the end. Claude shows a caution notice on any prompt that arrives through a link; that’s expected.

Across the IRW

Can item text predict difficulty across the IRW? compute.R takes each English dichotomous cognitive table with at least 200 respondents and item text in the IRW, and correlates word count and mean word length with the logit of proportion correct within the table. Pooling items across tables would confound an item’s length with the test it came from. No feature from a language model is used yet.

Recall from Constructs and construct maps: of the nine number-series items, NS_2 was solved by 84% of those who answered it and NS_6, whose key accepts 165 where tripling the steps gives 120, by 1.5%. The corpus run includes that table to check its own pipeline.

Code 
dd_path <- file.path("..", "deepdives", "ai-psychometrics", "results.rds")
dd <- if (file.exists(dd_path)) readRDS(dd_path) else NULL
if (!is.null(dd)) {
  ns <- dd$items[dd$items$table == "himmelstein-number_series-2025", ]
  print(round(setNames(ns$p, ns$item)[c("NS_2", "NS_6")], 3))   # the pipeline check
  s <- dd$summary[order(dd$summary$r_word_count), ]
  print(data.frame(table = s$table, items = s$n_items, median_p = round(s$median_p, 2),
                   r_word_count = round(s$r_word_count, 2), r_word_length = round(s$r_word_length, 2)),
        row.names = FALSE)
}
 NS_2  NS_6 
0.839 0.015 
                                table items median_p r_word_count r_word_length
                    preschool_sel_box    20     0.59        -0.62         -0.40
                rmet_higgins_2022_tom    16     0.55        -0.17          0.15
 himmelstein-impossible_question-2025    90     0.63        -0.11          0.15
       himmelstein-number_series-2025     9     0.48         0.07         -0.23
             gcbs_brotherton_2013_vcl    16     0.86           NA         -0.40
WarningPartial run: 5 of 33 tables

This is a pilot run of compute.R, on three tables whose answers are known (the number series, this lesson’s impossible questions, and preschool_sel_box, the IRW vignette’s strongest case) and 3 drawn at random, to check the pipeline. The full run replaces it.

Run on 25 September 2026: 5 tables summarised, 1 failed (polca_cheating: fewer than 5 items with both responses and text). Within-table correlations of word count with logit proportion correct run from -0.62 to 0.07 (median -0.14).

The pipeline reproduces the number-series proportions. In this pilot the correlations for word count are small, except for the preschool task at −0.62; the full run will show how they spread across all the candidate tables. The IRW’s item language and difficulty vignette runs the same analysis with a plot for each table.

Going further

  • Kapoor, R., Truong, S. T., Haber, N., Ruiz-Primo, M. A., & Domingue, B. W. (2025). Prediction of item difficulty for reading comprehension items by creation of annotated item repository. arXiv:2502.20663. https://arxiv.org/abs/2502.20663. Difficulty predicted from annotated features and language-model embeddings, for state reading items.
  • Benedetto, L., Cremonesi, P., Caines, A., Buttery, P., Cappelli, A., Giussani, A., & Turrin, R. (2023). A survey on recent approaches to question difficulty estimation from text. ACM Computing Surveys, 55(9), 1–37. https://doi.org/10.1145/3556538. The methods for predicting difficulty from text, sorted by the kind of question.
  • Gorin, J. S., & Embretson, S. E. (2006). Item difficulty modeling of paragraph comprehension items. Applied Psychological Measurement, 30(5), 394–411. https://doi.org/10.1177/0146621606288554. Difficulty explained by a cognitive model of the items.
  • Stenner, A. J., Burdick, H., Sanford, E. E., & Burdick, D. S. (2006). How accurate are Lexile text measures? Journal of Applied Measurement, 7(3), 307–322. https://pubmed.ncbi.nlm.nih.gov/16807496/. How precisely a readability framework measures texts.
  • Liu, Y., Bhandari, S., & Pardos, Z. A. (2025). Leveraging LLM respondents for item evaluation: A psychometric analysis. British Journal of Educational Technology, 56(3), 1028–1052. https://doi.org/10.1111/bjet.13570. Language models as respondents, judged against students.
  • Zhou, W., Liu, Y., Tang, R., & Wilson, M. (2026). Aligning LLM-simulated and human examinees for psychometric calibration: A cognitive diagnostic profiling approach. arXiv. https://arxiv.org/abs/2607.26317. Attribute profiles written as student descriptions, used to prompt language models to answer the fraction items (a preprint).
  • Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337–351. https://doi.org/10.1017/pan.2023.2. Simulated survey respondents conditioned on real backgrounds.
  • Scarlatos, A., Fernandez, N., Ormerod, C., Lottridge, S., & Lan, A. (2025). SMART: Simulated students aligned with item response theory for question difficulty prediction. arXiv:2507.05129. https://arxiv.org/abs/2507.05129. Simulated students trained to follow an item response model.
  • Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x. The kinds of evidence an automated scorer needs.
  • Lewis, J., & Casabianca, J. M. (2026). Limitations of QWK in evaluating automated and human scoring systems. Educational Measurement: Issues and Practice, 45(1). https://doi.org/10.1111/emip.70017. How QWK depends on the margins and the length of the scale.
  • McCaffrey, D. F., Casabianca, J. M., & Johnson, M. S. (2025). Measuring the accuracy of true score predictions for AI scoring evaluation. Journal of Educational Measurement, 62(4), 763–786. https://doi.org/10.1111/jedm.70011. PRMSE as an alternative to QWK.
  • Alvero, A. J., Giebel, S., Gebre-Medhin, B., Antonio, A. L., Stevens, M. L., & Domingue, B. W. (2021). Essay content and style are strongly related to household income and SAT scores: Evidence from 60,000 undergraduate applications. Science Advances, 7(42), eabi9031. https://doi.org/10.1126/sciadv.abi9031. What essays carry besides writing quality.
  • Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3). https://ejournals.bc.edu/index.php/jtla/article/view/1650. An automated essay scorer, described by its developers.
  • Embretson, S. E. (1999). Generating items during testing: Psychometric issues and models. Psychometrika, 64(4), 407–433. https://doi.org/10.1007/BF02294564. Items generated from a cognitive design, and models that calibrate the design.
  • Gierl, M. J., & Haladyna, T. M. (Eds.). (2012). Automatic item generation: Theory and practice. Routledge. https://doi.org/10.4324/9780203803912. Item models and templates.
  • Russell-Lasalandra, L. L., Christensen, A. P., & Golino, H. (2024). Generative psychometrics via AI-GENIE: Automatic item generation with network-integrated evaluation. PsyArXiv. https://doi.org/10.31234/osf.io/fgbj4. Items from language models, pruned by network psychometrics.
  • Haladyna, T. M., & Downing, S. M. (1993). How many options is enough for a multiple-choice test item? Educational and Psychological Measurement, 53(4), 999–1010. https://doi.org/10.1177/0013164493053004013. How often distractors perform.
  • Martínez-Plumed, F., Prudêncio, R. B. C., Martínez-Usó, A., & Hernández-Orallo, J. (2019). Item response theory in AI: Analysing machine learning classifiers at the instance level. Artificial Intelligence, 271, 18–42. https://doi.org/10.1016/j.artint.2018.09.004. Psychometrics turned on AI systems.
  • From the IRW: the item language and difficulty vignette, which this lesson’s deep dive follows.

Data sources

Data from the Item Response Warehouse (each table name links to its IRW page); references from IRW biblio.

For instructors

  • Source: EDUC 252 class 10 (slides c10, 1–41: the two intersections, automated scoring and QWK, simulated respondents, item difficulty modelling, automatic item generation) and c10/model.R in ben-domingue/252.
  • Timing: the core ideas and widgets take about 50 minutes; the real-data section is a 30-minute lab. The three glmer fits take about a minute in local R.
  • Changed from 252: the slides’ automated-scoring evidence is a widget, since no IRW table pairs human and engine scores. The slides’ remark on prediction metrics is worked out as the respondents yardstick and applied to the impossible questions. The meta-analysis QWK of about 0.8 (slides 7 and 16) and the human–human QWK of 0.60 (slide 17) are left out until their sources are verified. The slides’ state reading-item project is cited (Kapoor et al., 2025); their ENEM project and AI-generated reading-efficiency forms (slides 37–41) are left out.
  • Held back: rater effects for human and machine raters (Rater models); distractor analysis with the nominal model (Nominal response and multiple-choice models); predicted difficulties for new items in a bank (Item banks and adaptive testing); differential functioning of machine scores. Models as respondents is problem 6.
  • Across the IRW: deepdives/ai-psychometrics/compute.R is run by hand (it needs a Redivis token); the page reads its results.rds. The pilot took under a minute for six tables.
  • Solutions: held back in solutions/ai-psychometrics.qmd (not published). Problem 6 is open.