Morphology and function of the dorsolateral prefrontal cortex (dlPFC), and corresponding working memory performance, are affected early in the aging process, but nearly half of aged individuals are spared of working memory deficits. Translationally relevant model systems are critical for determining the neurobiological drivers of this variability. The common marmoset (Callithrix jacchus) is advantageous as a model for these investigations because, as a non-human primate, marmosets have a clearly defined dlPFC that enables measurement of prefrontal-dependent cognitive functions, and their short (∼10 year) lifespan facilitates longitudinal studies of aging. Previously, we characterized working memory capacity in a cohort of marmosets that collectively covered the lifespan, and found age-related working memory impairment. We also found a remarkable degree of heterogeneity in performance, similar to that found in humans. Here, we tested the hypothesis that changes to synaptic ultrastructure that affect synaptic efficacy stratify marmosets that age with cognitive impairment from those that age without cognitive impairment. We utilized electron microscopy to visualize synapses in the marmoset dlPFC and measured the sizes of boutons, presynaptic mitochondria, and synapses. We found that coordinated scaling of the sizes of synapses and mitochondria with their associated boutons is essential for intact working memory performance in aged marmosets. Further, lack of synaptic scaling, due to a remarkable failure of synaptic mitochondria to scale with presynaptic boutons, selectively underlies age-related working memory impairment. We posit that this decoupling results in mismatched energy supply and demand, leading to impaired synaptic transmission. We also found that aged marmosets have fewer synapses in dlPFC than young, though the severity of synapse loss did not predict whether aging occurred with or without cognitive impairment. This work identifies a novel mechanism of synapse dysfunction that stratifies marmosets that age with cognitive impairment from those that age without cognitive impairment. The process by which synaptic scaling is regulated is yet unknown and warrants future investigation.