Animal behavior is not driven simply by its current observa-tions, but is strongly influenced by internal states. Estimatingthe structure of these internal states is crucial for understand-ing the neural basis of behavior. In principle, internal statescan be estimated by inverting behavior models, as in inversemodel-based Reinforcement Learning. However, this requirescareful parameterization and risks model-mismatch to the ani-mal. Here we take a data-driven approach to infer latent statesdirectly from observations of behavior, using a partially ob-servable switching semi-Markov process. This process has twoelements critical for capturing animal behavior: it captures non-exponential distribution of times between observations, andtransitions between latent states depend on the animal’s actions,features that require more complex non-markovian models torepresent. To demonstrate the utility of our approach, we applyit to the observations of a simulated optimal agent performinga foraging task, and find that latent dynamics extracted by themodel has correspondences with the belief dynamics of theagent. Finally, we apply our model to identify latent states inthe behaviors of monkey performing a foraging task, and findclusters of latent states that identify periods of time consistentwith expectant waiting. This data-driven behavioral model willbe valuable for inferring latent cognitive states, and thereby formeasuring neural representations of those states.