There are many benefits to having a highly accurate representation of the environment. Why, then, has evolutionequipped us with a visual working memory (VWM) system that can represent only a handful of items with high accuracy?Here, we offer a normative explanation for this limitation by conceptualizing VWM as a system that balances between twoconflicting goals: keeping memory errors small and spiking activity low. We formalize this trade-off in a loss function andshow that minimization of loss dictates a strategy in which memory precision declines with the number of remembered items.Using psychophysical data from 67 human subjects in 5 delayed-estimation experiments, we show that this normative modelprovides an excellent account of human VWM limitations. These results suggest that human VWM implements an optimalcompromise between two conflicting ecological goals