In a noisy but structured world, memory can be improvedby enhancing limited stimulus-specific memory with statisti-cal information about the context. To do this, people have tolearn the statistical structure of their current environment. Wepresent a Sequential Monte Carlo (particle filter) model of howpeople track the statistical properties of the environment acrossmultiple contexts. This model approximates non-parametricBayesian clustering of percepts over time, capturing how peo-ple impute structure in their perceptual experience in order tomore efficiently encode that experience in memory. Each trialis treated as a draw from a context-specific distribution, wherethe number of contexts is unknown (and potentially infinite).The model maintains a finite set of hypotheses about how thepercepts encountered thus far are assigned to contexts, updat-ing these in parallel as each new percept comes in. We applythis model to a recall task where subjects had to recall the posi-tion of dots (Robbins, Hemmer, & Tang, 2014). Unbeknownstto subjects, each dot appeared in one of a few pre-defined re-gions on the screen. Our model captures subjects’ ability tolearn the inventory of contexts, the statistics of dot positionswithin each context, and the statistics of transitions betweencontexts—as reflected in both recall and prediction.