Planar black holes in AdS have long-lived quasinormal modes which capture the physics of charge and momentum diffusion in the dual field theory. How should we characterize the effective dynamics of a probe system coupled to the conserved currents of the dual field theory? Specifically, how would such a probe record the long-lived memory of the black hole and its Hawking fluctuations? We address this question by exhibiting a universal gauge invariant framework which captures the physics of stochastic diffusion in holography: a designer scalar with a gravitational coupling governed by a single parameter, the Markovianity index. We argue that the physics of gauge and gravitational perturbations of a planar Schwarzschild-AdS black hole can be efficiently captured by such designer scalars. We demonstrate that this framework allows one to decouple, at the quadratic order, the long-lived quasinormal and Hawking modes from the short-lived ones. It furthermore provides a template for analyzing fluctuating open quantum field theories with memory. In particular, we use this set-up to analyze the diffusive Hawking photons and gravitons about a planar Schwarzschild-AdS black hole and derive the quadratic effective action that governs fluctuating hydrodynamics of the dual CFT. Along the way we also derive results relevant for probes of hyperscaling violating backgrounds at finite temperature.