The excitonic insulator is an electronically-driven phase of matter that emerges upon the spontaneous formation and Bose condensation of excitons. Detecting this exotic order in candidate materials is a subject of paramount importance, as the size of the excitonic gap in the band structure establishes the potential of this collective state for superfluid energy transport. However, the identification of this phase in real solids is hindered by the coexistence of a structural order parameter with the same symmetry as the excitonic order. Only a few materials are currently believed to host a dominant excitonic phase, Ta$_2$NiSe$_5$ being the most promising. Here, we test this scenario by using an ultrashort laser pulse to quench the broken-symmetry phase of this transition metal chalcogenide. Tracking the dynamics of the material's electronic and crystal structure after light excitation reveals surprising spectroscopic fingerprints that are only compatible with a primary order parameter of phononic nature. We rationalize our findings through state-of-the-art calculations, confirming that the structural order accounts for most of the electronic gap opening. Not only do our results uncover the long-sought mechanism driving the phase transition of Ta$_2$NiSe$_5$, but they also conclusively rule out any substantial excitonic character in this instability.