Metal halide perovskites are promising candidates for next-generation photovoltaic and optoelectronic applications. The flexible nature of the octahedral network introduces complexity when understanding their physical behavior. It has been shown that these materials are prone to decomposition, phase competition, and the local crystal structure often deviates from the average space group symmetry. To make stable phase-pure perovskites, understanding their structure-composition relations is of central importance. We demonstrate, from lattice dynamics calculations, that the 24 inorganic perovskites ABX 3 (A = Cs, Rb; B = Ge, Sn, Pb; X = F, Cl, Br, I) exhibit instabilities in their cubic phase. These instabilities include cation displacements, octahedral tilting, and Jahn-Teller distortions. The magnitudes of the instabilities vary depending on the chemical identity and ionic radii of the composition. The tilting instabilities are energetically dominant, and reduce as the tolerance factor increases, whereas cation displacements and Jahn-Teller type distortions depend on the interactions between the constituent ions. We further considered representative tetragonal, orthorhombic and monoclinic perovskites phases to obtain phonon-stable phases for each composition. This work provides insights into the thermodynamic driving force of the instabilities and will help guide synthesis in material screening.