Collecting new information about the outside world is a key aspect of brain function. In the context of vision, we move our eyes multiple times per second to accumulate evidence about a scene. Prior studies have suggested that this process is goal-directed and close to optimal. Here, we show that this process of seeking new information suffers from a confirmation bias similar to what has been observed in a wide range of other contexts. We present data from a new gaze-contingent task that allows us to both estimate a participant's current belief, and compare that to their subsequent eye-movements. We find that these eye-movements are biased in a confirmatory way. Finally, we show that these empirical results can be parsimoniously explained under the assumption that the brain performs approximate, not exact, inference, with computations being more approximate in decision-making compared to sensory areas.