Human reinforcement learning (RL) is characterized by different challenges. Exploration has been studied extensively in multi-armed bandits, while planning has been investigated in multi-step decision tasks. More recent work has added structured rewards to study generalization. However, past studies have often focused on a single one of these aspects, making it hard to compare results. We propose a generative model for constructing correlated trees to provide a unified and scalable method for studying exploration, planning, and generalization in a single task. In an online experiment, we find that people use structure (when provided) to generalize and perform uncertainty-directed exploration, with structure helping more in larger environments. In environments without structure, exploration becomes more random and more planning is needed. All behavioral effects are captured in a single model with recoverable parameters. In conclusion, our results connect past research on human RL in one framework using correlated trees.

Theories of intrinsic motivation describe how behavior is driven by inherent satisfaction beyond only rewarding outcomes. One computational theory quantifies “fun” as the pleasure derived from improving one’s model of the environment. Here, we refine and test this theory by predicting maximal fun occurs when learning progress is also maximal, corresponding to a balance between ability (or knowledge) and task difficulty. Across multiple natural data sets (e.g., “Super Mario Maker”, “Trackmania”, or “Robozzle”), we confirm our prediction that human judgments of fun are highest at intermediate levels of difficulty. We provide further evidence through a number guessing experiment, where we manipulated the learnability of the environment by controlling the variance of the numbers that could be guessed. Both participants’ engagement and model-based analyses confirmed our predictions. Beyond simply exploiting maximal rewards or exploring maximal uncertainty, the constraints of learnability require a balance between challenge and ease.

Human cognition can tackle a wide range of problems, producing scalable results within a manageable timeframe. Many cognitive models can predict human behavior but lack such scalability, resulting in rapidly increasing processing times for more complex inputs. We present a task where participants mentally sort sequences of rectangles by size while we measure reaction times (RTs) and accuracy. By manipulating the size of the input and the presence of latent structure in the sequences, we investigate i) how mental sorting scales with input complexity, ii) how latent structure influences scaling, and iii) how mental computations can be captured by plausible cognitive models. Our results reveal RTs scale linearly with sequence length, and participants can learn and actively use latent structure to sort faster. This behavior is in line with a noisy sorting algorithm, which sequentially rules out potential hypotheses about the latent structure, thus reducing complexity while retaining accuracy.

Compositionality is a central component of the human faculty for generalization and flexibility. However, the computations involved are poorly understood, especially in terms of their cognitive costs. On one hand, compositionality requires searching combinatorially large hypothesis spaces, raising issues of tractability. On the other hand, compositional representations afford efficient and compact compression. To shed light on the cognitive resource required for compositionality, we used a within-subject time pressure manipulation to study how participants navigated a series of mazes, generated using recursive operations over spatial primitives. We find evidence that behavior is guided by the use of primitives and abstract operations over them, where the degree of compositional structure increases performance and speeds up decisions. And while time pressure led to more random errors, it did not impair the capacity for compositionality. Rather, participants increased their reliance on reusing and recombining previous computations, suggesting a remarkable robustness of human compositional reasoning.

Sensation-seeking (SS) is characterised by a proclivity for intense experiences and disregard for potential aversive consequences. While SS is implicated as a vulnerability factor in various mental disorders, the underlying mechanisms remain elusive. Recent approaches propose an alternative perspective, suggesting that SS may be linked to highly explorative, and therefore risky, behaviours driven by a preference for informative environments. To probe this hypothesis, we reanalysed a dataset where participants chose to self-administer or avoid mild electric stimulation (MES) in an economic decision-making task. Contrary to previous interpretations associating higher sensation-seeking with the positive economic value of experiencing MES, Bayesian models of learning reveal an alternative account: sensation-seekers are more attuned to information about stimuli-shock contingencies. Specifically, high sensation-seeking individuals are less avoidant of information about the possibility of a shock, supporting the idea that sensation-seeking is linked to a preference for informative environments.

Recent studies and models have been successful in describing human search in choice-rich environments in terms of different generalization and exploration strategies. However, it remains an open question how and to what extent people adapt these strategies to different environments. Therefore, we used a within-subject manipulation to study human search in two classes of environments, where the reward distributions differed in the strength of spatial correlations. These environments were chosen through simulations designed to maximize the need for deploying different generalization strategies. We describe participant behavior with the parameters of a model using Gaussian Process function learning to generalize and an Upper Confidence Bound to sample from unexplored options. Our results show that while participants adapted their exploration strategies, their generalization remained unchanged across environments. Together, these findings offer important first insights into how people adapt decision-making strategies depending on environmental demands in vast decision spaces.

Despite increased interest in creating more diverse and inclusive organizational environments, bias exists in how we choose leaders, who we interact with, and who we consider influential. Drawing from leadership emergence theory, we investigate potential interventions that support diverse leaders. Using agent-based simulations, we model a collective search process on a fitness landscape. Agents combine individual and social learning, and are represented as a feature vector blending relevant (e.g., individual learning characteristics) and irrelevant (e.g., race or gender) features. Agents use rational principles of learning to estimate feature weights on the basis of performance predictions, which are used to dynamically define social influence in their network. We show how biases arise based on historic privilege, but can be drastically reduced through the use of an intervention (e.g. mentorship). This framework allows us to test interventions best suited for unlearning bias in favor of performance-relevant traits.

Many aspects of human learning have been proposed as a process of constructing mental programs: from acquiring symbolic number representations to intuitive theories about the world. In parallel, there is a long-tradition of using information processing to model human cognition through Rate Distortion Theory (RDT). Yet, it is still poorly understood how to apply RDT when mental representations take the form of programs. In this work, we adapt RDT by proposing a three way trade-off among rate (description length), distortion (error), and computational costs (search budget). We use simulations on a melody task to study the implications of this trade-off, and show that constructing a shared program library across tasks provides global benefits. However, this comes at the cost of sensitivity to curricula, which is also characteristic of human learners. Finally, we use methods from partial information decomposition to generate training curricula that induce more effective libraries and better generalization.

Selecting informative queries is a crucial component of learning and decision-making, where models of information searchhave been widely used to provide normative guidance. Yet a typical requirement of these models is complete informationabout the underlying probabilistic structure of the environment, which is seldom met in real-world situations. Thus,information search models are blind to the epistemic uncertainty that comes with learning through experience, and do notdistinguish between probabilities estimated from a sample of two and a sample of one million. We develop a learningparadigm where a successful strategy needs to balance the exploration of queries with high epistemic uncertainty, with theexploitation of queries already known to be useful. We show that a Bayesian sampling variant of traditional informationsearch models learns faster and performs better, but most surprisingly, that a simple take-the-difference heuristic (TTD)performs competitively using only the absolute difference between observed frequencies.

Information sharing in competitive environments may seem counterintuitive, yet it is widely observed in humans and other animals. For instance, the open-source software movement has led to new and valuable technologies being released publicly to facilitate broader collaboration and further innovation. What drives this behavior and under which conditions can it be ben- eficial for an individual? Using simulations in both static and dynamic environments, we show that sharing information can lead to individual benefits through the mechanisms of pseudo- reciprocity, whereby shared information leads to by-product benefits for an individual without the need for explicit recipro- cation. Crucially, imitation with a certain level of innovation is required to avoid a tragedy of the commons, while the mecha- nism of a local visibility radius allows for the coordination of self-organizing collectives of agents. When these two mecha- nisms are present, we find robust evidence for the benefits of sharing—even when others do not reciprocate.