Psychometrics with the IRW: course map

DRAFT
This course is under construction and is not ready for teaching. Every lesson is a draft and will change, and nothing here has been checked by anyone but its authors. Please don't cite, assign, or circulate it yet. Comments are very welcome: please open an issue on GitHub, naming the lesson.

These are stand-alone lessons on psychometrics, each built around open data from the Item Response Warehouse. You can teach them in sequence as a full course or pick out individual lessons. The lessons come in three tranches:

Each lesson lists what it assumes and what builds on it. The lessons grew out of EDUC 252 (Introduction to Psychometrics) at Stanford. The source (lessons, code and data notes) is on GitHub; comments and issues are welcome there.

The lessons were written largely by Claude (Anthropic), an AI model, working from the EDUC 252 materials and under Ben Domingue’s direction. Ben reviews each lesson before it is marked done; until then, read its text, code and interpretations as a draft.

Why psychometrics?

Much of what we want to know about people hinges on attributes we cannot observe directly: how a child’s executive functioning changes with age, whether an intervention reduces anxiety, how reading develops across school. What we observe instead are responses to items: a tap on the correct arrow, “most days” on a survey, a wrong answer to question 7. Psychometrics is the discipline that turns those responses into measurements and tells us how far to trust them.

Consider a ruler. It measures a child’s height regardless of that child’s background; a different ruler gives a reading with the same meaning; and a well-chosen ruler measures quickly across the whole range of heights we care about. Psychological measures rarely do all of this, but the ruler suggests what to aim for (Domingue, 2026):

  • Measurements should be insensitive to differences in attributes other than the one being measured.
  • Measurements from different instruments should be comparable.
  • Measurement should deliver precise information about the attribute as quickly as possible.

None of these comes for free. Suppose we measure executive functioning with a tablet task. Children with less tablet experience may look worse for reasons that have nothing to do with executive functioning. The task that suits a four-year-old is not the one that suits a ten-year-old, yet we want to talk about growth across those ages. And time with a child is short, so every item has to earn its place. Psychometrics has tools for each problem: checking whether items work the same way across groups (Differential item functioning), putting scores from different forms on a common scale (Linking and equating), and choosing items to fit the respondent (Item banks and adaptive testing).

The common thread is a probabilistic model for how people respond to items. Such a model can be falsified, so we can interrogate whether it works in a given setting; when it does, it gives a compact description of how an instrument functions, one that can be reused with new people, new forms and new ages.

Psychological measurement is hard, and whether it deserves the name “measurement” at all has been argued over for most of a century (What is measurement?). Interval scales do not come easily (Scale properties). But the limits of our measures are limits on what we can learn about the attributes themselves. Psychometrics is how we find out where those limits are, and how we push them back.

Lessons by module

Lessons in blue have a draft to read. Lessons in grey are still stubs: a placeholder page with no content yet.

Foundations

  1. What is measurement? preliminary draft
  2. Item response data and the IRW preliminary draft
  3. The likelihood and logistic regression preliminary draft

Classical test theory

  1. Constructs and construct maps draft
  2. Classical test theory and reliability draft
  3. Where CTT breaks: toward item response models draft
  4. From construct map to items: building an instrument draft
  5. Many sources of error: generalizability theory draft

Factor analysis

  1. Factor analysis I: exploring dimensionality draft
  2. Factor analysis II: confirmatory models, omega, and IRT as factor analysis draft
  3. Measurement invariance: comparing groups with multigroup CFA draft
  4. Structural equation modeling with lavaan extension draft

Validity

  1. Validity as an argument draft
  2. Validity as causation: Borsboom’s view extension draft
  3. Gathering validity evidence draft

Item response theory

  1. The Rasch model draft
  2. Information, precision, and short forms draft
  3. Estimating abilities: MLE and EAP draft
  4. Estimating item parameters: JML and EM extension draft
  5. From the 1PL to the 4PL draft
  6. Parameter invariance extension draft
  7. Model fit and out-of-sample prediction draft
  8. Guessing and priors extension draft
  9. Dimensionality and multidimensional IRT extension draft
  10. Models for polytomous responses draft
  11. Person fit: does a respondent’s pattern fit the model? extension draft
  12. Missing responses: omitted, not reached, and ignorable extension draft
  13. Nonparametric IRT: Mokken scaling extension draft

Fairness and invariance

  1. Fairness in testing: bias, impact, and measurement invariance draft
  2. Differential item functioning draft
  3. Measurement invariance under treatment and life events extension draft

Using calibrated items

  1. Item banks and adaptive testing extension draft
  2. Linking and equating draft
  3. What does a score mean? Norms, error bands, and cut scores draft
  4. Scale properties: is the scale equal-interval? extension draft

Beyond the standard models

  1. What is an item? Explanatory item response models extension draft
  2. Latent classes and cognitive diagnosis extension draft
  3. IRTree models extension draft
  4. Unfolding models extension draft
  5. Nominal response and multiple-choice models extension draft
  6. Response time and the speed-accuracy tradeoff extension draft
  7. Trials as items: brightness, rotation, Stroop and shots extension draft
  8. AI and psychometrics extension draft
  9. Process models for response time: drift diffusion and race models extension draft
  10. Competitions and paired comparisons: Bradley-Terry and Elo extension draft
  11. Rater models: many-facet Rasch and beyond extension draft
  12. Continuous responses: sliders and bounded scales extension draft
  13. Mixture models: when respondents differ in kind extension draft

Build your own course

Build a course from the lesson bank: drag lessons into your course (or use Add), put them in any order, and group them under unit or week headings. Start from a preset or from nothing. Nothing is blocked. The notes flag a lesson that comes before a lesson it assumes, or that assumes one you left out, and offer a fix you can take or ignore. Open “Dependencies and threads” under a lesson in the bank to see what it needs, what it leads to, and the threads it starts and picks up. The lessons, prerequisites and threads all come from lessons.yml.

Paths through the course

A path is one order for teaching the lessons. The builder’s “A first course” and “EDUC 252” presets start from these lessons. Add your own path to lessons.yml.

A first course (preliminary and core lessons)

  1. What is measurement?
  2. Item response data and the IRW
  3. The likelihood and logistic regression
  4. Constructs and construct maps
  5. Classical test theory and reliability
  6. From construct map to items: building an instrument
  7. Many sources of error: generalizability theory
  8. Where CTT breaks: toward item response models
  9. Validity as an argument
  10. Gathering validity evidence
  11. Factor analysis I: exploring dimensionality
  12. The Rasch model
  13. From the 1PL to the 4PL
  14. Information, precision, and short forms
  15. Estimating abilities: MLE and EAP
  16. Factor analysis II: confirmatory models, omega, and IRT as factor analysis
  17. Measurement invariance: comparing groups with multigroup CFA
  18. Model fit and out-of-sample prediction
  19. Models for polytomous responses
  20. Fairness in testing: bias, impact, and measurement invariance
  21. Differential item functioning
  22. What does a score mean? Norms, error bands, and cut scores
  23. Linking and equating

EDUC 252 (Stanford), a 10-week course

  1. What is measurement?
  2. Item response data and the IRW
  3. The likelihood and logistic regression
  4. Constructs and construct maps
  5. Classical test theory and reliability
  6. Where CTT breaks: toward item response models
  7. Validity as an argument
  8. The Rasch model
  9. From the 1PL to the 4PL
  10. Information, precision, and short forms
  11. Estimating abilities: MLE and EAP
  12. Validity as causation: Borsboom’s view
  13. Parameter invariance
  14. Dimensionality and multidimensional IRT
  15. Fairness in testing: bias, impact, and measurement invariance
  16. Differential item functioning
  17. Estimating item parameters: JML and EM
  18. Model fit and out-of-sample prediction
  19. Guessing and priors
  20. Linking and equating
  21. Measurement invariance under treatment and life events
  22. What is an item? Explanatory item response models
  23. Latent classes and cognitive diagnosis
  24. Models for polytomous responses
  25. Item banks and adaptive testing
  26. IRTree models
  27. Unfolding models
  28. Nominal response and multiple-choice models
  29. Response time and the speed-accuracy tradeoff
  30. Trials as items: brightness, rotation, Stroop and shots
  31. AI and psychometrics