Do Individual Effects Reflect Quantitative or Qualitative Differences in Cognition?

Rouder and Haaf (2020) posed the important question if there are some individuals whose behavior is not in accordance with well-established experimental effects and whether these individual differences are quantitative or qualitative in nature. In our commentary, we discuss the distinction between quantitative and qualitative individual differences and between individual and average causal effects and come to the conclusion that this is not a new question, but in fact one that has already been discussed by Gordon W. Allport (1937) and Donald B. Rubin (1974, 1978). Moreover, we critically examine their proposed rule of thumb to collect about 100 trials per experimental condition to reliably measure individual differences in typical experimental effects. Based on simulation results, we suggest to not rely on any general rule of thumb, but to use simulation studies and the convenient quid function provided by the authors to make more informed decisions regarding trial numbers for specific experimental designs.