Drude oscillator potentials are a popular and computationally efficient class of polarizable models that represent each polarizable atom as a positively charged Drude core harmonically bound to a negatively charged Drude shell. We show that existing force fields that place all non-Coulomb forces on the Drude core and none on the shell inadvertently couple the dipole to non-Coulombic forces. This introduces errors where interactions with neutral particles can erroneously induce atomic polarization, leading to spurious polarizations in the absence of an electric field, exacerbating violations of equipartition in the employed Carr-Parinello scheme. A suitable symmetrization of the interaction potential that correctly splits the force between the Drude core and shell can correct this shortcoming, improving the stability and numerical performance of Drude oscillator-based simulations. The symmetrization procedure is straightforward and only requires the rescaling of a few force field parameters.