DIF with several person factors and repeated measures
Josh McGrane
Source:vignettes/dif-repeated-measures.Rmd
dif-repeated-measures.RmdDesign the analysis around the person
Differential item functioning is a violation of the Rasch model’s requirement of invariant comparison: item locations should not depend on which persons respond (Rasch 1961). When several person factors are relevant, they should enter one model. Repeated observations require a further distinction: group is a between-person factor, whereas occasion varies within person.
For a factor \(G\), class interval \(C\), and standardised residual \(z\), the single-factor model is
\[ z=\mu+G+C+G\mathbin{:}C+\varepsilon. \]
The \(G\) term tests uniform DIF.
The \(G\mathbin{:}C\) term tests
non-uniform DIF. With several factors, dif_anova fits their
terms jointly and uses Type II sums of squares.
The following dataset has two observations per person. Item I03 has a group shift, item I06 has an occasion shift, and item I05 shifts for group B at the second occasion only, a group-by-occasion interaction.
set.seed(21)
N <- 320
difficulty <- seq(-1.5, 1.5, length.out = 8)
theta <- rnorm(N)
group <- rep(c("A", "B"), each = N / 2)
make_wave <- function(occasion_shift, interaction_shift) {
shift <- matrix(0, N, 8)
shift[group == "B", 3] <- 1.2
shift[, 6] <- occasion_shift
shift[group == "B", 5] <- interaction_shift
matrix(rbinom(N * 8, 1,
plogis(outer(theta, difficulty, "-") - shift)), N, 8)
}
X <- rbind(make_wave(0, 0), make_wave(1.0, 2.0))
colnames(X) <- sprintf("I%02d", 1:8)
dat <- data.frame(
pid = rep(sprintf("P%03d", seq_len(N)), 2),
X,
group = rep(group, 2),
occasion = rep(c("T1", "T2"), each = N)
)Fit once and test both factors
The repeated person identifier is carried into the fit.
dif_anova detects occasion as within-person; it can also be
declared explicitly. Persons, not stacked rows, are the units of
analysis. Uniform between-person terms use Type II tests with HC3
covariance. Class-interval interactions retain the residual-ANOVA
reference. Within-person tests use person-level contrasts, with a
Greenhouse–Geisser correction when a factor has more than two levels.
This mixed-design analysis extends the single-factor residual analysis
of variance described by Andrich and Marais (2019). Its F references are
large-sample approximations.
fit <- rasch(dat, id = "pid", factors = c("group", "occasion"),
items = sprintf("I%02d", 1:8))
da <- dif_anova(fit, within = "occasion", effects = "factorial", sizes = TRUE)
da$summary
#> item term F_uniform p_uniform p_uniform_adj eta2_uniform uniform_DIF
#> I01 group 0.786 0.376 1.000 0.003
#> I01 occasion 0.875 0.350 1.000 0.003
#> I01 group:occasion 3.153 0.077 1.000 0.010
#> I02 group 0.287 0.593 1.000 0.001
#> I02 occasion 4.885 0.028 1.000 0.016
#> I02 group:occasion 0.809 0.369 1.000 0.003
#> I03 group 15.018 < 0.001 0.006 0.048 *
#> I03 occasion 1.340 0.248 1.000 0.004
#> I03 group:occasion 2.787 0.096 1.000 0.009
#> I04 group 1.292 0.257 1.000 0.004
#> I04 occasion 1.327 0.250 1.000 0.004
#> I04 group:occasion 1.821 0.178 1.000 0.006
#> I05 group 15.045 < 0.001 0.006 0.051 *
#> I05 occasion 10.604 0.001 0.055 0.033
#> I05 group:occasion 27.711 < 0.001 < 0.001 0.083 *
#> I06 group 6.249 0.013 0.544 0.020
#> I06 occasion 19.306 < 0.001 < 0.001 0.059 *
#> I06 group:occasion 2.340 0.127 1.000 0.008
#> I07 group 4.837 0.029 1.000 0.015
#> I07 occasion 4.613 0.033 1.000 0.015
#> I07 group:occasion 2.761 0.098 1.000 0.009
#> I08 group 1.259 0.263 1.000 0.004
#> I08 occasion 0.491 0.484 1.000 0.002
#> I08 group:occasion 1.593 0.208 1.000 0.005
#> F_nonuniform p_nonuniform p_nonuniform_adj eta2_nonuniform nonuniform_DIF
#> 0.212 0.932 1.000 0.003
#> 1.165 0.326 1.000 0.015
#> 1.051 0.381 1.000 0.014
#> 3.400 0.010 0.416 0.042
#> 0.814 0.517 1.000 0.010
#> 1.631 0.166 1.000 0.021
#> 2.355 0.054 1.000 0.030
#> 2.148 0.075 1.000 0.027
#> 0.421 0.793 1.000 0.005
#> 0.513 0.726 1.000 0.007
#> 1.739 0.141 1.000 0.022
#> 1.083 0.365 1.000 0.014
#> 2.483 0.044 1.000 0.031
#> 0.812 0.518 1.000 0.010
#> 2.916 0.022 0.886 0.037
#> 1.294 0.272 1.000 0.017
#> 1.333 0.257 1.000 0.017
#> 1.000 0.408 1.000 0.013
#> 0.558 0.693 1.000 0.007
#> 0.645 0.631 1.000 0.008
#> 1.597 0.175 1.000 0.020
#> 1.514 0.198 1.000 0.019
#> 0.386 0.818 1.000 0.005
#> 0.886 0.472 1.000 0.011
#> superseded
#>
#>
#>
#>
#>
#>
#>
#>
#>
#>
#>
#>
#> *
#> *
#>
#>
#>
#>
#>
#>
#>
#>
#>
#> The multiplicity adjustment covers the complete family of
item-by-DIF-term tests. Uniform DIF is a factor effect that is stable
over the trait; a factor-by-class-interval effect is non-uniform DIF.
effects = "factorial" adds the person-factor interactions.
A significant higher-order term supersedes its component group terms
within the same item: item I05’s group effect is significant on its own,
but the superseded flag records that the interaction
absorbs it, and the follow-ups report the interaction rather than its
components. Read adjusted probabilities with effect sizes before
changing an item.
Quantify the departure
ANOVA identifies evidence against invariance; it does not state the
size of the departure in logits. dif_size resolves an item
by a between-person factor. dif_contrasts provides planned
contrasts and uses person-level differencing for within-person
questions.
dif_size(fit, "I03", by = "group")
#> DIF size for I03 by group (resolved locations, logits)
#> level location se weak n
#> A -0.518 0 320
#> B 0.227 0 320
#> level_a level_b difference se z p p_adj lower upper significant practical ets
#> A B -0.745 >= 0.50 <NA>
#> signed_area
#>
#> p adjusted by holm over 1 pairwise comparison(s); practical criterion 0.50 logits
#> notes: person identifiers repeat across response rows: resolved point differences remain descriptive, but sampling SEs, confidence intervals and Wald tests are withheld; use dif_contrasts for person-level inference or a whole-person bootstrap
dc <- dif_contrasts(fit, items = c("I03", "I06"), within = "occasion")
dc$table
#> item contrast within estimate se statistic df
#> I03 group: B - A 0.737 4.044 316.916
#> I03 occasion: T2 - T1 * -0.258 -0.876 313.935
#> I03 group(B - A) x occasion(T2 - T1) * -0.258 -1.397 313.935
#> I06 group: B - A -0.434 -2.138 308.822
#> I06 occasion: T2 - T1 * 0.906 4.438 298.364
#> I06 group(B - A) x occasion(T2 - T1) * 0.134 1.220 298.364
#> p p_adj lower upper significant practical
#> < 0.001 < 0.001 * *
#> 0.382 0.490
#> 0.163 0.490
#> 0.033 0.133
#> < 0.001 < 0.001 * *
#> 0.223 0.490
da$posthoc
#> item term item.1 contrast within estimate se statistic
#> I03 group I03 B - A 0.737 4.044
#> I05 group:occasion I05 B - A x T2 - T1 * 2.140 5.137
#> I06 occasion I06 T2 - T1 * 0.906 4.438
#> df p p_adj lower upper significant practical
#> 316.916 < 0.001 < 0.001 * *
#> 317.916 < 0.001 < 0.001 * *
#> 298.364 < 0.001 < 0.001 * *dif_posthoc() is the general follow-up for a significant
term. A main effect with more than two levels is reported as pairwise
marginal differences over the other fitted factors. An interaction is
reported as a difference-in-differences, or its higher-order
counterpart. These comparisons use the joint covariance of the resolved
item locations and Holm adjustment over the stated family. The result is
on the logit scale and respects the factor structure used in the DIF
analysis. Here item I03 is reported as a pairwise group difference, item
I06 as a person-level occasion contrast, and item I05 as the
difference-in-differences of its interaction; the superseded I05 group
term receives no follow-up of its own.
For repeated-person contrasts, significance comes from person-level
residual contrast scores with the same design-cell weights as the
resolved estimate. Other fitted factors are averaged equally over their
complete cells, including when their sample sizes differ, and the
independent between-person cells use a Welch–Satterthwaite reference.
The resolved logit difference remains the magnitude, but its
row-independent calibration covariance is not a repeated-measures
standard error; the package therefore withholds the logit SE and
interval in this case. The fitted person identifier is used unless
id is supplied explicitly.
For a many-facet fit, follow-ups may name the underlying item; its virtual facet cells are pooled with common weights so facet severity cancels from the group contrast. Extended-frame fits support the residual ANOVA for factors outside the frame definition, but not an ordinary resolved-item magnitude: that refit would discard the fitted frame units.
A statistical flag should be considered with the logit magnitude, targeting, item content, and the intended use of the scale. Resolving an item changes the measurement model and should follow a substantive account of why the item is not invariant.
References
Andrich, D., and Marais, I. (2019). A Course in Rasch Measurement Theory: Measuring in the Educational, Social and Health Sciences. Springer.
Rasch, G. (1961). On general laws and the meaning of measurement in psychology. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability (Vol. 4, pp. 321–333). Berkeley: University of California Press.