Childhood adversity (e.g., poverty, violence exposure) has been associated with broad cognitive deficits as well as with cognitive adaptations in specific abilities. Integrating these perspectives requires a process-level understanding of how deficit and adaptation processes operate. We investigated how adversity was associated with inhibition, attention shifting, and mental rotation in the Adolescent Brain Cognitive Development (ABCD) study (N ≈ 10,500). Using Hierarchical Bayesian Drift Diffusion Modeling, we distinguished between speed of information uptake, response caution, and stimulus encoding/response execution. We further used structural equation modeling to isolate task-general and task-specific variances in each of these processing stages. Youth with more exposure to household threat showed slower task-general processing speed, but showed intact task-specific abilities. In addition, youth with more exposure to household threat and material deprivation tended to respond more cautiously in general. These findings suggests that traditional assessments might overestimate the extent to which childhood adversity reduces specific abilities.