The average and local structure of the oxides Ba$_2$SiO$_4$, BaAl$_2$O$_2$, SrAl$_2$O$_4$, and Y$_2$SiO$_5$ are examined in order to evaluate crystal rigidity in light of recent studies suggesting that highly connected and rigid structures yield the best phosphor hosts. Simultaneous momentum-space refinements of synchrotron X-ray and neutron scattering yield accurate average crystal structures, with reliable atomic displacement parameters. The Debye temperature $\Theta_{\mathrm{D}}$, which has proven to be a useful proxy for structural rigidity, is extracted from the experimental atomic displacement parameters and compared with predictions from density functional theory calculations and experimental low-temperature heat capacity measurements. The role of static disorder on the measured displacement parameters, and the resulting Debye temperatures are also analyzed using pair distribution function of total neutron scattering, as refined over varying distance ranges of the pair distribution function. The interplay between optimal bonding in the structure, structural rigidity, and correlated motion in these structures is examined, and the different contributions are delineated.