To what extent do human reward learning and decision-making rely on the ability to represent and generate richly structured relationships between options? We provide evidence that structure learning and the principle of compositionality play crucial roles in human reinforcement learning. In a new multi-armed bandit paradigm, we found evidence that participants are able to learn representations of different reward structures and combine them to make correct generalizations about options in novel contexts. Moreover, we found substantial evidence that participants transferred knowledge of simpler reward structures to make compositional generalizations about rewards in complex contexts. This allowed participants to accumulate more rewards earlier, and to explore less whenever such knowledge transfer was possible. We also provide a computational model which is able to generalize and compose knowledge for complex reward structures. This model describes participant behaviour in the compositional generalization task better than various other models of decision-making and transfer learning.

How do humans generalise to make better decisions? Previous work has investigated this question using reward-guided decision-making tasks with low-dimensional and artificial stimuli. In this paper, we extend this work by presenting participants with a naturalistic decision-making task, in which options were images of real-world objects and the underlying reward function was based on one of their latent dimensions. Even though participants received no explicit instruction about object features, they quickly learned to do the task and generalised to unseen objects. To understand how they accomplished this, we tested a range of computational models and found that human behaviour is overall best explained by a linear model but that participants' strategies changed during the experiment. Lastly, we showed that combining pixel-based representations extracted from convolutional neural networks with the original latent dimensions further improved our models. Taken together, our study offers new insights into human decision making under naturalistic settings.