Controlling a phase of matter by coherently manipulating specific vibrational modes has long been an attractive (yet elusive) goal for ultrafast science. Solids with strongly correlated electrons, in which even subtle crystallographic distortions can result in colossal changes of the electronic and magnetic properties, could be directed between competing phases by such selective vibrational excitation. In this way, the dynamics of the electronic ground state of the system become accessible, and new insight into the underlying physics might be gained. Here we report the ultrafast switching of the electronic phase of a magnetoresistive manganite via direct excitation of a phonon mode at 71 meV (17 THz). A prompt, five-order-of-magnitude drop in resistivity is observed, associated with a non-equilibrium transition from the stable insulating phase to a metastable metallic phase. In contrast with light-induced, and current-driven phase transitions, the vibrationally driven bandgap collapse observed here is not related to hot-carrier injection and is uniquely attributed to a large-amplitude Mn-O distortion. This corresponds to a perturbation of the perovskite-structure tolerance factor, which in turn controls the electronic bandwidth via inter-site orbital overlap. Phase control by coherent manipulation of selected metal--oxygen phonons should find extensive application in other complex solids--notably in copper oxide superconductors, in which the role of Cu-O vibrations on the electronic properties is currently controversial.