P-type point contact (PPC) HPGe detectors are a leading technology for rare event searches due to their excellent energy resolution, low thresholds, and multi-site event rejection capabilities. We have characterized a PPC detector's response to $\alpha$ particles incident on the sensitive passivated and p+ surfaces, a previously poorly-understood source of background. The detector studied is identical to those in the MAJORANA DEMONSTRATOR experiment, a search for neutrinoless double-beta decay ($0 u\beta\beta$) in $^{76}$Ge. $\alpha$ decays on most of the passivated surface exhibit significant energy loss due to charge trapping, with waveforms exhibiting a delayed charge recovery (DCR) signature caused by the slow collection of a fraction of the trapped charge. The DCR is found to be complementary to existing methods of $\alpha$ identification, reliably identifying $\alpha$ background events on the passivated surface of the detector. We demonstrate effective rejection of all surface $\alpha$ events (to within statistical uncertainty) with a loss of only 0.2% of bulk events by combining the DCR discriminator with previously-used methods. The DCR discriminator has been used to reduce the background rate in the $0 u\beta\beta$ region of interest window by an order of magnitude in the MAJORANA DEMONSTRATOR, and will be used in the upcoming LEGEND-200 experiment.