How much can we infer about the past and future, given our knowledge of the present? Unlike temporally symmetric inferences about simple sequences, inferences about our own lives are asymmetric: we are better able to infer the past than the future, since we remember our past but not our future (i.e., the psychological arrow of time). What happens when both the past and future are unobserved, as when we make inferences about other people’s lives? We had participants view segments of a character-driven television drama. They wrote out what would happen just before or after each just-watched segment. Participants were better at inferring past (versus future) events. This asymmetry was driven by participants’ reliance on characters’ conversational references in the narrative, which tended to favor the past. Our work reveals a temporal asymmetry in how observations of other people’s behaviors can inform us about the past and future.