IRTree models

Module: Beyond the standard models
Tranche: extension (beyond a first course)
Before this: Models for polytomous responses · What is an item? Explanatory item response models
Builds toward: none
Picks up threads: Keying has to be checked; the data can’t always tell (from Classical test theory and reliability)
IRW tables: bfi_goldberg_1992_conscientiousness, introversion_extroversion
From EDUC 252: slides c8; problem sets PS8#4; code c8/irtree.R, c8/irtree2.R, c8/irtree_sim.R, ps8/fast.R
Status: draft
Authorship: Written largely by Claude (Anthropic), from Ben Domingue’s EDUC 252 materials and under Ben’s direction. Ben reviews each lesson before it is marked done.
Comments: open an issue on GitHub about this lesson.

What this is for

What does a 3 mean on a scale from 1 (disagree) to 5 (agree)? The graded response model reads it as a little more of the trait than a 2. But a respondent may choose the middle because they have no view, or prefer not to commit, and a 5 may say as much about a taste for the ends of a scale as about the trait. An IRTree writes the response as a sequence of decisions (take a side? which side? how far?) and gives each its own item response model and, if we want, its own \(\theta\). Models for polytomous responses split a response into binary questions fixed by the model; a tree chooses the questions to match a process we have in mind, and fitting it is a recoding followed by a glmer() fit from What is an item?. On ten conscientiousness items, leaving the middle and going to the ends turn out to be largely one tendency, which barely tracks conscientiousness.

Goals

By the end of this lesson you should be able to:

  • write a rating as a path through a tree of binary nodes, and its probability as a product along the path;
  • choose a tree’s shape (linear or nested) and its abilities (one \(\theta\), or one per node), and say which choice gives the sequential model;
  • recode long response data into pseudo-items and fit a tree with glmer();
  • separate content from midpoint and extreme responding, and say what the separation assumes;
  • use a first node for speed or skipping, and say what it protects the content estimate from.

Core ideas

A response as a sequence of decisions

Recall from Models for polytomous responses: ordinal models split a response into binary questions. The sequential model (Tutz, 1990) asks “having reached category \(k-1\), does the respondent go on to \(k\)?”, and a response answers only the questions it reaches.

Nothing requires the questions to follow the order of the categories. Böckenholt (2012) proposed a tree for a five-point rating that asks three different questions. Does the respondent take a side or choose the middle? Which side? And do they go to its end? Each question is a node, and each node gets an item response model. With a Rasch model at every node, \[p^{(n)}_{ij} = \Pr(\text{node } n = 1) = \frac{\exp\big(\theta^{(n)}_j - b^{(n)}_i\big)}{1 + \exp\big(\theta^{(n)}_j - b^{(n)}_i\big)}, \qquad n \in \{\text{side}, \text{direction}, \text{extreme}\},\] and the probability of a response is the product of the node probabilities along its path. A 4 takes a side, the agree side, and stops short of the end: \[\Pr(x_{ij} = 4) = p^{(\text{side})}_{ij}\; p^{(\text{direction})}_{ij}\,\big(1 - p^{(\text{extreme})}_{ij}\big), \qquad \Pr(x_{ij} = 3) = 1 - p^{(\text{side})}_{ij}.\] Each response becomes one binary pseudo-item per node it reaches; a node it doesn’t reach is absent, not a 0 (De Boeck & Partchev, 2012).

Try it: build a tree

Pick a response and a tree; the highlighted path is the one it takes.

Under the nested tree a 3 is one pseudo-item and every other response three; under the linear tree a 1 is one and a 4 or 5 four. Each response follows exactly one path, so the five probabilities sum to one (problem 1).

Tree shape and node abilities are choices

Two choices define a tree. Its shape: a linear tree asks its questions in a chain, as the sequential model does; a nested tree branches, so that different responses pass through different nodes, as Böckenholt’s does (De Boeck & Partchev, 2012). And its abilities: one \(\theta\) shared by every node, or a correlated \(\theta\) per node. A linear tree with one \(\theta\) and Rasch steps is Tutz’s sequential model. Jeon and De Boeck (2016) extend trees further, with slopes at each node and more flexible links between nodes and dimensions.

Try it: node abilities and the category curves

Ten items have node difficulties \(b^{(\text{side})} = -1.2\), \(b^{(\text{direction})} = -0.9\) and \(b^{(\text{extreme})} = 0.5\), close to the averages of the conscientiousness items below. The horizontal axis is the content \(\theta\), the one at the direction node. Set the other two, or tie all three together.

With a \(\theta\) per node, the curve for 3 is flat (0.23 at the defaults): choosing the middle depends only on the side node. Lower that \(\theta\) and the curve rises everywhere; raise the extreme node’s and the 1s and 5s take over from the 2s and 4s. Now tie the nodes together: the respondents lowest on the trait choose 3 most (0.69 at \(\theta = -2\)), because the \(\theta\) that pushes them toward disagreeing also keeps them from taking a side. A shared \(\theta\) assumes every node moves along one scale, as a linear tree’s steps do; in a nested tree it forces different questions onto one dimension.

Fitting a tree is recoding, then a mixed model

Recall from What is an item?: the Rasch model is a mixed logistic regression, glmer(resp ~ 0 + item + (1 | id), family = binomial), with one intercept per item (its easiness, \(-b_i\)) and a random intercept per respondent (\(\theta_j\)). Item features and respondent covariates are changes to that formula.

How does a rating become pseudo-items? Take three respondents from the conscientiousness data in With real data and their answers to the first two items, C1 and C2. (C2 is worded against conscientiousness, so its responses are reversed first, and higher means more conscientious on both.) Böckenholt’s tree can be written as a map with one row per category and one column per node. Each row is that category’s path, with NA where the path doesn’t reach a node, and recoding is looking each response up in the map:

Code 
# Three respondents from the sample drawn below (rows 1, 2 and 6) and their keyed
# answers to C1 and C2.
ex <- data.frame(id = c(1, 1, 2, 2, 6, 6), item = c("C1", "C2"), x = c(3, 5, 4, 4, 1, 2))
# The tree as a map: one row per category, one column per node. Each row is that
# category's path; NA marks a node the path doesn't reach.
tree_map <- rbind("1" = c(side = 1, direction = 0,  extreme = 1),
                  "2" = c(side = 1, direction = 0,  extreme = 0),
                  "3" = c(side = 0, direction = NA, extreme = NA),
                  "4" = c(side = 1, direction = 1,  extreme = 0),
                  "5" = c(side = 1, direction = 1,  extreme = 1))
# Recoding is looking each response up in the map: one pseudo-item per node.
cbind(ex, tree_map[ex$x, ], row.names = NULL)
id item x side direction extreme
1 C1 3 0 NA NA
1 C2 5 1 1 1
2 C1 4 1 1 0
2 C2 4 1 1 0
6 C1 1 1 0 1
6 C2 2 1 0 0
Code 
# Long: one row per response and node, then drop the nodes a response doesn't reach.
long_ex <- data.frame(ex[rep(seq_len(nrow(ex)), each = 3), c("id", "item")],
                      node = colnames(tree_map), resp = c(t(tree_map[ex$x, ])), row.names = NULL)
long_ex[!is.na(long_ex$resp), ]
id item node resp
1 1 C1 side 0
4 1 C2 side 1
5 1 C2 direction 1
6 1 C2 extreme 1
7 2 C1 side 1
8 2 C1 direction 1
9 2 C1 extreme 0
10 2 C2 side 1
11 2 C2 direction 1
12 2 C2 extreme 0
13 6 C1 side 1
14 6 C1 direction 0
15 6 C1 extreme 1
16 6 C2 side 1
17 6 C2 direction 0
18 6 C2 extreme 0

Each response becomes one pseudo-item per node. Respondent 1’s 3 on C1 is side = 0 and nothing else. Respondent 6’s 1 and 2 differ only at the extreme node, and respondent 2’s two 4s are coded alike. The long format, one row per response and node reached, drops the NAs: six responses become 16 rows (rows 2 and 3 were respondent 1’s NAs). Those rows are the data the model is fitted to.

Try it: recode one respondent

Set one respondent’s answers to three items; they start at respondent 1’s keyed answers to C1, C2 and C3. The first table is the respondent’s row with one column per node and item, the layout mirt takes; the second is the same codes stacked long, the layout glmer takes.

Each 3 puts two NAs in the wide row and takes two rows out of the long table; every other answer gives three rows. Changing a 4 to a 5 changes one code, at the extreme node, and changing a 2 to a 4 changes one, at the direction node.

A tree is one of those regressions. With the long rows as the data (pseudo_items),

glmer(resp ~ 0 + item:node + (0 + node | id), family = binomial, data = pseudo_items)

gives every item an easiness at every node (item:node) and every respondent a \(\theta\) at every node, with their SDs and correlations estimated ((0 + node | id)). The node is an item predictor, and it has to be a factor: stored as a number, glmer would treat it as one slope. One \(\theta\) for all nodes is (1 | id). Any software for binary item response models can fit a tree, because the likelihood factors into one piece per node (Go deeper, below); mirt (Chalmers, 2012) takes the pseudo-items as columns, missing where a response doesn’t reach a node.

Assumptions: a tree with nodes \(n = 1, \dots, N\); each response category corresponds to exactly one path; at each node, the probability of a 1 depends only on the respondent’s node ability and the item’s node parameters; given the node abilities, the node outcomes are independent, within and across items.

One response. Write \(y_{ijn} \in \{0, 1\}\) for the pseudo-item of respondent \(j\), item \(i\) and node \(n\), and \(r_{ijn} = 1\) if the response reaches node \(n\) (0 otherwise). The path of \(x_{ij}\) fixes both, so \[\Pr(x_{ij} \mid \boldsymbol\theta_j) = \prod_{n} \Big[\big(p^{(n)}_{ij}\big)^{y_{ijn}} \big(1 - p^{(n)}_{ij}\big)^{1 - y_{ijn}}\Big]^{r_{ijn}},\] where a node the response doesn’t reach contributes a factor of 1.

All responses. Multiplying over items and respondents, \[L = \prod_j \int \prod_n \prod_{i:\, r_{ijn} = 1} \big(p^{(n)}_{ij}\big)^{y_{ijn}} \big(1 - p^{(n)}_{ij}\big)^{1 - y_{ijn}}\; \phi(\boldsymbol\theta_j; \mathbf 0, \boldsymbol\Sigma)\, d\boldsymbol\theta_j.\] Given \(\boldsymbol\theta_j\), this is the likelihood of an ordinary binary item response model for a data set with one column per item and node, and missing entries where \(r_{ijn} = 0\). Whether a node is reached is fixed by the outcomes of earlier nodes, which are observed, so the missing pseudo-items are missing at random in the sense of Rubin (1976), and leaving them out loses nothing. That is why recoding followed by glmer() or mirt fits the tree, with no special software.

Two consequences. If \(\boldsymbol\Sigma\) is diagonal (node abilities uncorrelated), the integral splits and the tree is \(N\) separate item response models, one per node. And if all nodes share one \(\theta\) and the tree is linear with Rasch nodes, the product is the sequential model of Models for polytomous responses, whose steps Tutz (1990) fits the same way.

Response styles

If you’ve done From construct map to items: acquiescence, agreeing whatever the statement, is one reason scales include reversed items. Once the reversed items are keyed, acquiescence pulls the two halves of the sum in opposite directions and largely cancels.

Acquiescence is one response style, a tendency in how a respondent uses the scale. Two others are midpoint responding, choosing the middle, and extreme responding, choosing the ends (Baumgartner & Steenkamp, 2001). Reversed items don’t balance these two, since reversing maps 5 to 1 and 3 to 3. In Böckenholt’s tree they have their own nodes. The \(\theta\) at the side node is a respondent’s tendency to leave the middle; the \(\theta\) at the extreme node is their tendency to go to the ends; and the content \(\theta\), at the direction node, is estimated net of both (Böckenholt, 2017). Khorramdel and von Davier (2014) fit style dimensions shared across all five Big Five scales.

The separation rests on assumptions. The tree takes a 3 to carry no information about direction, and takes the side and extreme decisions to be related to content only through the correlations of the \(\theta\)s. Plieninger and Meiser (2014) checked the style dimensions against external measures of response style and found that they behaved as style; Meiser, Plieninger and Henninger (2019) relax the first assumption with nodes that are themselves ordinal.

Try it: same conscientiousness, different styles

Two respondents with the same content \(\theta\) answer the ten items above. A leaves the middle and goes to the ends; B does neither. The lines are their expected sum scores.

At the defaults A expects 43.0 and B 35.1, from the same conscientiousness. The lines cross at \(-0.9\), the direction node’s difficulty, and below it A scores lower than B. Extreme responding stretches sums away from the middle in whichever direction the content points, so no constant can correct for it.

Trees for what the response hides: speed and skips

If you’ve done Guessing and priors: in a timed lexical decision task, responses under 0.3 seconds were correct at about chance, and a few respondents gave most of them. Rapid guessing there looked like a property of respondents, not items. Skipped responses were left for this lesson.

A first node can record what the response doesn’t show. A node for “answered in under \(t\) seconds” gives every respondent a speed \(\theta\), correlated with the content \(\theta\), and the content node can then be fitted to slow responses only, or given separate item parameters for fast and slow ones (Partchev & De Boeck, 2012; DiTrapani, Jeon, De Boeck & Partchev, 2016). Time as a continuous variable is the subject of Response time and the speed-accuracy tradeoff.

A skip is usually scored as wrong or left missing, both reasonable when skipping has nothing to do with ability. When lower-ability respondents skip more, skipping carries information about \(\theta\), and the missing responses are not ignorable (Holman & Glas, 2005; Glas & Pimentel, 2008). A tree whose first node is “answered?” and second “correct, given answered?” puts that information into the model: the correlation between the two node \(\theta\)s carries it to the content estimate (Debeer, Janssen & De Boeck, 2017). None of the IRW tables with a tokenless download has enough skips to show this, so the simulation below does.

Simulate

Part 1 generates 1–5 ratings from Böckenholt’s tree (400 respondents, 10 items, correlated node \(\theta\)s), fits one \(\theta\) and a \(\theta\) per node, and compares both with the truth. Part 2 simulates a 15-item right–wrong test on which lower-ability respondents skip more, and scores the skips as wrong, as missing, and with a skip node. The first run takes one to two minutes, most of it installing and loading lme4; nAGQ = 0 keeps each fit to seconds.

With the seed as written, AIC prefers a \(\theta\) per node (11,234 against 11,679), and the node SDs and correlations come back within about 0.13 of the truth. The content estimates from the per-node tree correlate 0.88 with the true content \(\theta\), against 0.69 from the one-\(\theta\) tree, which mixes content with style. In part 2, the 77 respondents who skipped most skipped about half the items. Scored as wrong, their abilities come out 0.24 too low on average; left missing, 0.41 too high, since the model pulls respondents with few answers toward the mean, and these respondents are below it. With the skip node the error is 0.02. Try R with every correlation set to 0, a skip rate that doesn’t depend on ability (skipper <- rnorm(np2)), or np <- 2000 locally.

Download this code to run it as large as you like in your own R session.

With real data

Conscientiousness and response styles

The main example is bfi_goldberg_1992_conscientiousness: the ten conscientiousness items of the IPIP Big-Five factor markers (Goldberg, 1992; Goldberg et al., 2006), answered online around 2012 through the Open-Source Psychometrics Project. Its job is to be a five-point scale on which response styles could be at work. The IRW’s item-text check confirmed that these are the IPIP markers rather than the Big Five Inventory. IPIP items are in the public domain, so we can quote them: C1 is “I am always prepared” and C2 “I leave my belongings around” (Goldberg et al., 2006; IPIP key). Responses run from 1 (disagree) through 3 (neutral) to 5 (agree), with 2 and 4 unlabelled. Download the code to run every chunk yourself.

Recall from Classical test theory and reliability: items worded against the construct have to be reverse-keyed before they are combined, and the key should come from the documentation and be checked in the data, since neither is enough alone.

In a tree, keying decides what the direction node means: keyed, it measures conscientiousness on every item; unkeyed, it asks “agree?”, which mixes conscientiousness with acquiescence. The side and extreme nodes don’t change. The IPIP key reverses C2, C4, C6 and C8, and the data agree:

Code 
library(lme4)
options(digits = 7)  # R's default, in case a .Rprofile changes it
set.seed(51)
url <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v60_0.bfi_goldberg_1992_conscientiousness/rows?format=csv"
df <- read.csv(url)
df <- df[!is.na(df$resp), ]
wide <- tapply(df$resp, list(df$id, df$item), function(x) x[1])
wide <- wide[complete.cases(wide), paste0("C", 1:10)]
c(respondents = nrow(wide), items = ncol(wide))
respondents       items 
      19718          10
Code 
# The responses arrive as given: 1 = disagree, 3 = neutral, 5 = agree. C2, C4, C6
# and C8 are worded against conscientiousness. Before keying, their correlations
# with C1 ("I am always prepared") are negative:
round(cor(wide)[1, ], 2)
   C1    C2    C3    C4    C5    C6    C7    C8    C9   C10 
 1.00 -0.26  0.30 -0.35  0.39 -0.31  0.31 -0.32  0.43  0.33
Code 
# Key them (reversing maps k to 6 - k), so that higher = more conscientious.
reversed <- paste0("C", c(2, 4, 6, 8))
wide[, reversed] <- 6 - wide[, reversed]
round(cor(wide)[1, ], 2)
  C1   C2   C3   C4   C5   C6   C7   C8   C9  C10 
1.00 0.26 0.30 0.35 0.39 0.31 0.31 0.32 0.43 0.33
Code 
# A random 3,000 respondents keep the fits quick.
x <- wide[sample(nrow(wide), 3000), ]
round(prop.table(table(x)), 3)   # share of responses in each category
x
    1     2     3     4     5 
0.095 0.164 0.241 0.282 0.218

Before keying, the four reversed items correlate from \(-0.26\) to \(-0.35\) with C1; after, every item correlates positively with it (0.26 to 0.43). All 19,718 respondents answered all ten items. We take a random 3,000, whose responses lean to the agree side: 21.8% of responses are 5 and 9.5% are 1. Now the recoding:

Code 
# One row per node a response reaches. Node 1: took a side (1) or chose 3 (0).
# Given a side, node 2: the agree side, 4 or 5 (1), or the disagree side (0); and
# node 3: an extreme response, 1 or 5 (1), or a moderate one (0).
long <- data.frame(id = rep(seq_len(nrow(x)), ncol(x)),
                   item = rep(colnames(x), each = nrow(x)), x = c(x))
sided <- long[long$x != 3, ]
tree <- rbind(
  data.frame(long,  node = "side",      resp = as.integer(long$x != 3)),
  data.frame(sided, node = "direction", resp = as.integer(sided$x > 3)),
  data.frame(sided, node = "extreme",   resp = as.integer(sided$x %in% c(1, 5))))
tree$node <- factor(tree$node, levels = c("side", "direction", "extreme"))  # a factor, not text
tree$item <- factor(tree$item, levels = colnames(x))
# Rows 1, 2 and 6 of the sample: the answers in the recoding example.
x[c(1, 2, 6), 1:2]
      C1 C2
18087  3  5
13139  4  4
7362   1  2
Code 
# Respondent 1's pseudo-items for C1 and C2:
tree[tree$id == 1 & tree$item %in% c("C1", "C2"), c("item", "x", "node", "resp")]
item x node resp
1 C1 3 side 0
3001 C2 5 side 1
30011 C2 5 direction 1
30012 C2 5 extreme 1
Code 
table(tree$node)   # rows per node: every response reaches node 1

     side direction   extreme 
    30000     22762     22762

Respondent 1’s 3 on C1 becomes one row; their keyed 5 on C2 (a raw 1) becomes three. All 30,000 responses reach the side node, and the 22,762 that took a side reach the other two. Now the two trees:

Code 
# The same tree with one theta for all three nodes, and with a theta per node.
# Each item has its own easiness at each node (item:node); b = -easiness.
# nAGQ = 0 skips glmer's slowest step; the estimates barely move.
one <- glmer(resp ~ 0 + item:node + (1 | id), data = tree, family = binomial, nAGQ = 0)
per <- glmer(resp ~ 0 + item:node + (0 + node | id), data = tree, family = binomial, nAGQ = 0)
AIC(one, per)
df AIC
one 31 85738.29
per 36 82597.71

A \(\theta\) per node improves the AIC by more than 3,000 points (82,598 against 85,738, for five more parameters). The three decisions are not one dimension. Are they three unrelated ones?

Code 
VarCorr(per)   # SDs of the three node thetas and their correlations
 Groups Name          Std.Dev. Corr        
 id     nodeside      0.77931              
        nodedirection 1.74973  0.327       
        nodeextreme   1.23659  0.680 0.241

The side and extreme \(\theta\)s correlate 0.68: respondents who leave the middle also go to the ends. With conscientiousness they correlate only 0.33 and 0.24. The content \(\theta\) varies most (SD 1.75, against 0.78 and 1.24).

Code 
# What do the styles do to the sum score? theta estimates (conditional modes) for
# each respondent, split by where they stand on conscientiousness.
th <- ranef(per)$id
r <- rowSums(x)
high <- th$nodedirection > 1
low <- th$nodedirection < -1
round(c(sum_with_content = cor(r, th$nodedirection),
        sum_with_extreme_if_high = cor(r[high], th$nodeextreme[high]),
        sum_with_extreme_if_low = cor(r[low], th$nodeextreme[low])), 2)
        sum_with_content sum_with_extreme_if_high  sum_with_extreme_if_low 
                    0.95                     0.88                    -0.55
Code 
c(high = sum(high), low = sum(low))
high  low 
 825  785

The sum score and the content \(\theta\) correlate 0.95, so the sum orders respondents much as the content \(\theta\) does. But among the 825 respondents whose content \(\theta\) is above 1, the sum correlates 0.88 with the extreme \(\theta\), and among the 785 below \(-1\), \(-0.55\). As in the widget, extreme responding stretches the sum away from the middle: conscientious respondents who like the ends score higher than those who don’t, and unconscientious ones lower. (These are estimates of each respondent’s \(\theta\)s, shrunk toward the mean, so treat the correlations as descriptive.)

Can the tree’s AIC be compared with the graded response model’s? Only when both are computed the same way from the same responses. glmer with nAGQ = 0 approximates the marginal likelihood and mirt integrates it by quadrature, so the same tree gets a different log likelihood from each; and a tree that drops categories, as the speed tree below drops the content of fast responses, describes different data. Böckenholt’s tree describes all five categories, so we fit it and the GRM in mirt:

Code 
# Can the tree's AIC be compared with the GRM's? Fit both by the same method: mirt,
# which integrates over theta by quadrature. The tree becomes 30 pseudo-items (NA
# where a response doesn't reach a node), each loading on its node's dimension with
# a slope fixed at 1; the three dimensions have free variances and correlations.
pseudo <- data.frame(ifelse(x == 3, 0, 1),
                     ifelse(x == 3, NA, 1 * (x > 3)),
                     ifelse(x == 3, NA, 1 * (x == 1 | x == 5)))
names(pseudo) <- paste0(rep(c("side_", "dir_", "ext_"), each = 10), colnames(x))
spec <- mirt::mirt.model("side = 1-10\n dir = 11-20\n ext = 21-30\n COV = side*dir*ext")
tree_mirt <- mirt::mirt(pseudo, spec, itemtype = "Rasch", verbose = FALSE)
tree_mirt1 <- mirt::mirt(pseudo, 1, itemtype = "Rasch", verbose = FALSE)  # one theta
grm <- mirt::mirt(as.data.frame(x), 1, itemtype = "graded", verbose = FALSE)
fits <- list(tree_mirt1, tree_mirt, grm)
data.frame(parameters = c(attr(logLik(one), "df"), attr(logLik(per), "df"),
                          sapply(fits, mirt::extract.mirt, "nest")),
           logLik = round(c(logLik(one), logLik(per), sapply(fits, mirt::extract.mirt, "logLik"))),
           AIC = round(c(AIC(one), AIC(per), sapply(fits, mirt::extract.mirt, "AIC"))),
           row.names = c("tree, one theta, glmer", "tree, theta per node, glmer",
                         "tree, one theta, mirt", "tree, theta per node, mirt", "GRM, mirt"))
parameters logLik AIC
tree, one theta, glmer 31 -42838 85738
tree, theta per node, glmer 36 -41263 82598
tree, one theta, mirt 31 -42819 85699
tree, theta per node, mirt 36 -41144 82361
GRM, mirt 50 -41571 83242

If you’ve done Model fit and out-of-sample prediction: AIC compares models fitted to the same data by the same likelihood; it says which model is preferred, not how much better it predicts. The IMV puts that on a common scale.

The per-node tree’s log likelihood is \(-41{,}263\) from glmer and \(-41{,}144\) from mirt: 119 apart from the approximation alone. Within mirt, that tree beats the GRM (AIC 82,361 against 83,242, with 36 parameters against 50), and the tree with one \(\theta\) falls far behind both (85,699). The tree wins because it has style dimensions the GRM lacks (problem 3 gives a rival the same advantage).

What does the choice rest on? An assumption about the response process: that choosing the middle, a side and the end are separate decisions. When such a process is in view (midpoint avoidance, extreme responding, speed, skipping), the tree gives it a parameter and estimates the content net of it; the one-\(\theta\) tree, which drops that, was the worst of the three here. AIC says which model is preferred, not how much better each predicts. Comparing the trees and their rivals on held-out responses with the IMV (Model fit and out-of-sample prediction) puts that on a common scale, and it is a comparison you can run for any set of trees on any rating scale.

Extraversion and fast responses

For contrast, introversion_extroversion: 91 statements from the development of the Multidimensional Introversion-Extraversion Scales, answered online around 2012 through the same project (MIES development data and codebook), with the time spent on each. Its job is to ask whether fast responding says anything about the trait. Items 81–85 and 89–91 are eight of the ten IPIP extraversion markers (Goldberg, 1992), the same family as the conscientiousness items, and we use those. Responses run from 1 (disagree) to 5 (agree). Five of the eight are worded toward introversion (“I keep in the background”, for one; MIES codebook), so we key them, higher meaning more extraverted.

Code 
url_ie <- "https://redivis.com/api/v1/tables/datapages.item_response_warehouse:v60_0.introversion_extroversion/rows?format=csv"
ie <- read.csv(url_ie)
c(respondents = length(unique(ie$id)), items = length(unique(ie$item)))
respondents       items 
       7188          91
Code 
round(quantile(ie$rt, c(0.1, 0.5, 0.9)), 1)   # seconds per item, all 91 items
10% 50% 90% 
1.8 3.2 6.9
Code 
# Items 81-85 and 89-91 are eight of the ten IPIP extraversion markers. 81-85 are
# worded toward introversion, so we key them: higher = more extraverted.
e <- ie[ie$item %in% c(81:85, 89:91), ]
e$x <- ifelse(e$item %in% 81:85, 6 - e$resp, e$resp)
e$fast <- as.integer(e$rt < 2)
round(c(share_fast = mean(e$fast),
        median_rt = median(e$rt)), 2)
share_fast  median_rt 
      0.21       2.77
Code 
round(tapply(e$x == 3, e$fast, mean), 3)   # share choosing 3, slow (0) and fast (1)
    0     1 
0.157 0.082

The median response took 3.2 seconds across all 91 items and 2.77 on our eight, where 21% came in under 2 seconds; fast responses chose the middle less often (8.2% against 15.7%). A two-node tree on a random 1,500 respondents asks: was the response fast? If not, was it on the extraverted side?

Code 
# Two nodes, after ps8/fast.R: was the response fast (under 2 seconds)? If not,
# was it on the extraverted side (keyed 4 or 5)? Fast responses stop at node 1.
ids <- sample(unique(e$id), 1500)
s <- e[e$id %in% ids, ]
slow <- s[s$fast == 0, ]
st <- rbind(data.frame(id = s$id, item = s$item, node = "fast", resp = s$fast),
            data.frame(id = slow$id, item = slow$item, node = "trait",
                       resp = as.integer(slow$x >= 4)))
st$node <- factor(st$node, levels = c("fast", "trait"))
st$item <- factor(st$item)
speed <- glmer(resp ~ 0 + item:node + (0 + node | id), data = st, family = binomial, nAGQ = 0)
VarCorr(speed)
 Groups Name      Std.Dev. Corr   
 id     nodefast  1.4853          
        nodetrait 1.7392   -0.075

Respondents differ in speed (SD 1.49 at the fast node), but the speed and extraversion \(\theta\)s correlate \(-0.075\): how quickly someone answers these items says next to nothing about how extraverted they are. Are fast responses careless, then?

Code 
# Do fast responses carry the trait? Each respondent's mean keyed response on their
# fast items against the mean on their slow items.
m_fast <- tapply(e$x[e$fast == 1], e$id[e$fast == 1], mean)
m_slow <- tapply(e$x[e$fast == 0], e$id[e$fast == 0], mean)
both <- intersect(names(m_fast), names(m_slow))
c(respondents_with_both = length(both), r = round(cor(m_fast[both], m_slow[both]), 2))
respondents_with_both                     r 
              4668.00                  0.73

No. For the 4,668 respondents with both fast and slow responses, the mean keyed response on the fast items correlates 0.73 with the mean on the slow ones. Rapid responses in Guessing and priors were at chance; here a response under 2 seconds looks like a respondent who knew their answer.

Problems

  1. The tree’s probabilities. For Böckenholt’s tree, write \(\Pr(x = k)\) for \(k = 1, \dots, 5\) in terms of the three node probabilities, and show that they sum to 1. Then show that a linear tree with Rasch nodes and one \(\theta\) is the sequential model, and explain why it is not the partial credit model, although both have one parameter per category boundary.
  2. The other four traits. Fit Böckenholt’s tree to the other four bfi_goldberg_1992_* tables (key each from the IPIP key, checked in the data). Are the side and extreme \(\theta\)s as strongly correlated for every trait? Do one respondent’s style \(\theta\)s from different traits correlate, as a style should? (The five tables share respondent ids.)
  3. Comparing the tree with the GRM. When can you compare the tree’s AIC with the GRM’s? Then give a rival without a tree the same advantage: in mirt, a multidimensional GPCM whose second dimension scores categories 1 and 5 as 1 and the others as 0 (the gpcm_mats argument) has an extreme-response dimension of its own. Compare it with the tree. What does the comparison say about the tree’s advantage over the GRM?
  4. Two countries. A survey is given in two countries whose respondents differ in how often they choose the ends of a scale. What can Böckenholt’s tree correct for when you compare the countries’ means, and what can’t it? Would you change the instrument as well as the model?
  5. The 2-second cut. Refit the speed tree with cuts at 1, 1.5 and 3 seconds. How does the share of fast responses change, and the correlation between speed and extraversion? Is there a cut below which fast responses stop tracking the trait? (PS8#4)
  6. Challenge (open): the skip \(\theta\). On a low-stakes test, a skip node gives every respondent a second \(\theta\), their tendency to answer. Does it belong in the reported score? Consider what the score is for and whether a respondent could raise it by skipping less. I don’t have a general answer.

Stuck, or curious about something this lesson doesn’t cover? Ask Claude about this lesson opens a new conversation that already knows what the lesson is about; add your question at the end. Claude shows a caution notice on any prompt that arrives through a link; that’s expected.

Going further

  • De Boeck, P., & Partchev, I. (2012). IRTrees: Tree-based item response models of the GLMM family. Journal of Statistical Software, 48(Code Snippet 1). https://doi.org/10.18637/jss.v048.c01. The recoding into pseudo-items and the glmer fit, with linear and nested trees.
  • Böckenholt, U. (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17(4), 665–678. https://doi.org/10.1037/a0028111. The side, direction and extreme tree for rating scales.
  • Böckenholt, U. (2017). Measuring response styles in Likert items. Psychological Methods, 22(1), 69–83. https://doi.org/10.1037/met0000106. Trees for response styles, set beside other approaches.
  • Jeon, M., & De Boeck, P. (2016). A generalized item response tree model for psychological assessments. Behavior Research Methods, 48(3), 1070–1085. https://doi.org/10.3758/s13428-015-0631-y. Trees with slopes and more flexible links between nodes and dimensions.
  • Plieninger, H., & Meiser, T. (2014). Validity of multiprocess IRT models for separating content and response styles. Educational and Psychological Measurement, 74(5), 875–899. https://doi.org/10.1177/0013164413514998. Whether the style nodes measure style.
  • Meiser, T., Plieninger, H., & Henninger, M. (2019). IRTree models with ordinal and multidimensional decision nodes for response styles and trait-based rating responses. British Journal of Mathematical and Statistical Psychology, 72(3), 501–516. https://doi.org/10.1111/bmsp.12158. Ordinal nodes, relaxing the assumption that a midpoint says nothing about direction.
  • Khorramdel, L., & von Davier, M. (2014). Measuring response styles across the Big Five: A multiscale extension of an approach using multinomial processing trees. Multivariate Behavioral Research, 49(2), 161–177. https://doi.org/10.1080/00273171.2013.866536. Style dimensions shared across personality scales (problem 2).
  • Baumgartner, H., & Steenkamp, J.-B. E. M. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840. Response styles across countries, and what they do to means and correlations (problem 4).
  • Debeer, D., Janssen, R., & De Boeck, P. (2017). Modeling skipped and not-reached items using IRTrees. Journal of Educational Measurement, 54(3), 333–363. https://doi.org/10.1111/jedm.12147. Skip and not-reached nodes in achievement tests.
  • Holman, R., & Glas, C. A. W. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1–17. https://doi.org/10.1348/000711005X47168. Missingness as a second latent dimension, the idea behind a skip node.
  • Glas, C. A. W., & Pimentel, J. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68(6), 907–922. https://doi.org/10.1177/0013164408315262. The same idea for items not reached.
  • Partchev, I., & De Boeck, P. (2012). Can fast and slow intelligence be differentiated? Intelligence, 40(1), 23–32. https://doi.org/10.1016/j.intell.2011.11.002. Fast and slow responses as branches of a tree.
  • DiTrapani, J., Jeon, M., De Boeck, P., & Partchev, I. (2016). Attempting to differentiate fast and slow intelligence: Using generalized item response trees to examine the role of speed on intelligence tests. Intelligence, 56, 82–92. https://doi.org/10.1016/j.intell.2016.02.012. A speed node with its own dimension.
  • Tutz, G. (1990). Sequential item response models with an ordered response. British Journal of Mathematical and Statistical Psychology, 43(1), 39–55. https://doi.org/10.1111/j.2044-8317.1990.tb00925.x. The sequential model, a linear tree.
  • Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. https://doi.org/10.1093/biomet/63.3.581. Missing at random, and when missingness can be ignored.
  • Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1). https://doi.org/10.18637/jss.v067.i01. The package for the tree fits.
  • Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6). https://doi.org/10.18637/jss.v048.i06. The package for the tree and GRM fits compared by AIC.
  • Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006). The International Personality Item Pool and the future of public-domain personality measures. Journal of Research in Personality, 40(1), 84–96. https://doi.org/10.1016/j.jrp.2005.08.007. The public-domain item pool behind both examples.

Data sources

Data from the Item Response Warehouse (each table name links to its IRW page); references from IRW biblio.

For instructors

  • Source: EDUC 252 class 8 (slides c8, 31–42), problem set 8 (#4, fast responding), and the code in ben-domingue/252: c8/irtree.R, c8/irtree2.R, c8/irtree_sim.R and ps8/fast.R.
  • Timing: the core ideas and widgets take about 45 minutes; the conscientiousness analysis is a 20-minute lab and the speed tree another 10. The real-data code takes about three minutes in local R, most of it the mirt fit of the tree, so run it before class.
  • Changed from 252: the slides and PS8#4 use ffm_CSN, which has no tokenless download. The response-style analysis moves to bfi_goldberg_1992_conscientiousness (IPIP markers, like the ffm items) and the fast-responding analysis to eight IPIP extraversion markers in introversion_extroversion, which records response times. As in c8/irtree2.R, the fits use a random subsample. The keying uses the published IPIP key, checked in the data, rather than the sign of GRM slopes. The skipped-response example is simulated, since no tokenless IRW table has enough skips. The AIC comparison with the GRM is made in mirt, where both likelihoods are computed the same way.
  • Held back: response time as a continuous variable (Response time and the speed-accuracy tradeoff); whether style dimensions threaten a validity argument (not taken up in Gathering validity evidence, which comes earlier); choosing among trees for the same categories beyond AIC, for which the IMV is the tool.
  • Solutions: held back in solutions/irtrees.qmd (not published). Problem 6 is open.