The quantum nature of light makes it a basic component for models of quantum measurement and information exchange between disparate quantum modes, pioneered in the field of cavity quantum electrodynamics. The interaction of atomic ensembles with the mode of an optical cavity provides a flexible platform for exploring the coherent interaction of light with diverse macroscopic dynamics, such as collective motion and spin. This dissertation presents experimental results and theoretical models for continuous measurement and control of the center of mass motion and collective spin precession of an atomic ensemble, mediated by coupling to a high-finesse optical cavity. First, the theory of dispersive coupling between the cavity mode and the collective motion and spin of an atomic ensemble is derived, and then a general time-domain formalism is developed for theoretical analysis of multi-mode optodynamical systems. Single-mode optodynamical effects are introduced through experimental demonstrations of measurement and control of the collective atomic spin, providing a close analogy to cavity optomechanics.
Next, multiple collective atomic modes are considered within a single cavity, in order to assemble optically mediated interactions within multi-mode optodynamical systems. A demonstration of optodynamical interactions between the center of mass motion of two atomic ensembles is presented, coupled through an optical spring mediated by the cavity mode. Then simultaneous coupling of the center of mass motion and total spin precession of a single ensemble of atoms is described, yielding an experimental realization of a negative-mass instability, facilitated by the novel resource of the spin ensembles inverted state. A theoretical analysis of the negative-mass instability is presented, which indicates the possibility of generating two-mode squeezed states in the absence of excess incoherent noise. Finally, linear state retrodiction from the optodynamical signals is discussed, providing background and supplemental material for a forthcoming manuscript.