Stochastic events in our daily environments, such as a missed bus or forgotten keys, require adaptive understanding for efficient exploitation of the environment. In this study we tested how humans acquire and use such understanding if the either the type or probability of events change. We asked 281 participants to predict the location of an animated fly, which hid from the observers. Over the first 10 trials we induced a strong prior model of the task environment and subsequently introduced stochastic changes and new content to manipulate the rate of model violations. Prediction errors derived from a specific world model drove information seeking actions, leading to new explanations and associated probability estimates. Current world model constrained possible updates, often only leading to partial investigation and suboptimal strategies, especially when behavior had positive utility. Additionally, evidence for accurate understanding but failure to identify and exploit ideal behavior was a characteristic result.