Measurements show fast ion losses correlated with applied three-dimensional (3D) fields in a variety of plasmas ranging from L-mode to resonant magnetic perturbation (RMP) edge localized mode (ELM) suppressed H-mode discharges. In DIII-D L-mode discharges with a slowly rotating n = 2 magnetic perturbation, scintillator detector loss signals synchronized with the applied fields are observed to decay within one poloidal transit time after beam turnoff indicating they arise predominantly from prompt loss orbits. Full orbit following using M3D-C1 calculations of the perturbed fields and kinetic profiles reproduce many features of the measured losses and points to the importance of the applied 3D field phase with respect to the beam injection location in determining the overall impact on prompt beam ion loss. Modeling of these results includes a self-consistent calculation of the 3D perturbed beam ion birth profiles and scrape-off-layer ionization, a factor found to be essential to reproducing the experimental measurements. Extension of the simulations to full slowing down timescales, including fueling and the effects of drag and pitch angle scattering, show the applied n = 3 RMPs in ELM suppressed H-mode plasmas can induce a significant loss of energetic particles from the core. With the applied n = 3 fields, up to 8.4% of the injected beam power is predicted to be lost, compared to 2.7% with axisymmetric fields only. These fast ions, originating from minor radii ρ > 0.7, are predicted to be primarily passing particles lost to the divertor region, consistent with wide field-of-view infrared periscope measurements of wall heating in n = 3 RMP ELM suppressed plasmas. Edge fast ion Dα (FIDA) measurements also confirm a large change in edge fast ion profile due to the n = 3 fields, where the effect was isolated by using short 50 ms RMP-off periods during which ELM suppression was maintained yet the fast ion profile was allowed to recover. The role of resonances between fast ion drift motion and the applied 3D fields in the context of selectively targeting regions of fast ion phase space is also discussed.