The effect of defect chemistry on the polar and nonpolar phases of hexagonal InMnO3 is investigated using first-principles density functional calculations. Our motivation is to show how point defects and substitutional atoms can modify the delicate balance between ferroelectric and nonferroelectric phases in a complex multiferroic oxide. By analyzing the distinct In corrugation patterns of the competing phases, we find that oxygen interstitials, indium vacancies and indium-oxygen vacancy pairs favor the polar P63cm phase, which is also the ground state of stoichiometric InMnO3. The polar P3c1 phase is stabilized by oxygen vacancies, while Ga substitution on the Mn site destabilize the ferroelectric phases and favor instead the nonpolar P3c or P63/mmc structures. In addition to the structure, the electrical properties are also strongly dependent on the defect chemistry, ranging from metallic to large band gap insulating. The implications of the strong influence of vacancies, interstitials and substitutions on the ferroelectric and electronic properties are discussed with respect to synthesis and applications.