Despite the long-standing recognition that individuals vary in their cognitive performance across relatively short time periods, little research has integrated an understanding of short-term within-individual variation in cognitive performance into our theories of cognitive ability. We contend that systematic patterns of between-individual differences in within-individual variation are meaningful and should not be viewed merely as measurement error. We argue that predominant cognitive testing methods using between-individual analysis of single-occasion scores are limited in their capacity to develop a process account of why individuals with the same test score differ in practical contexts. We propose that short-term repeated measures paradigms (e.g., the Experience Sampling Method) be used to understand the nature and sources of between-individual differences in within-individual variation. Finally, we outline considerations for researchers when adapting this paradigm for cognitive assessment and present initial findings from our lab on the feasibility of this paradigm.

Cognitive flexibility is the ability to adapt to changing contingencies and demands over time. It is often assessed by complex problem solving tasks (e.g., microworld simulations), which require individuals to maintain a goal state over trials. Yet typically analyses summarises performance by mean accuracy per person, which assumes performance is a constant and ignores the trial-by-trial trajectory of individuals that reveal strategy . We tested n = 83 psychology students aged 16 to 45 in a microworld task under different conditions of stochasticity and delay. We modelled trial-by-trial performance according to a prospective model of the true task contingencies versus a reactive strategy adapting to feedback from the last trial,. Most individuals (~66%) appeared to adopt a reactive strategy, however when the task contingencies were immediate, people were more likely to act prospectively. Performance in the microworld task depends on the strategy which varies by person and with the task contingencies.