Observation of large nonsaturating magnetoresistance in rare-earth monopnictides has raised enormous interest in understanding the role of its electronic structure. Here, by a combination of molecular-beam epitaxy, low-temperature transport, angle-resolved photoemission spectroscopy, and hybrid density functional theory we have unveiled the band structure of LuSb, where electron-hole compensation is identified as a mechanism responsible for large magnetoresistance in this topologically trivial compound. In contrast to bulk single crystal analogues, quasi-two-dimensional behavior is observed in our thin films for both electron and holelike carriers, indicative of dimensional confinement of the electronic states. Introduction of defects through growth parameter tuning results in the appearance of quantum interference effects at low temperatures, which has allowed us to identify the dominant inelastic scattering processes and elucidate the role of spin-orbit coupling. Our findings open up possibilities of band structure engineering and control of transport properties in rare-earth monopnictides via epitaxial synthesis.

Topological materials often exhibit remarkably linear nonsaturating magnetoresistance (LMR), which is both of scientific and technological importance. However, the role of topologically nontrivial states in the emergence of such a behavior has eluded clear demonstration in experiments. Here, by reducing the coupling between the topological surface states (TSS) and the bulk carriers, we controllably tune the LMR behavior in Pt1-xAuxLuSb into distinct plateaus in Hall resistance, which we show arise from a quantum Hall phase. This allowed us to reveal how smearing of the Landau levels, which otherwise gives rise to a quantum Hall phase, results in an LMR behavior due to strong interaction between the TSS with a positive g factor and the bulk carriers. We establish that controlling the coupling strength between the surface and the bulk carriers in topological materials can bring about dramatic changes in their magnetotransport behavior. In addition, our work outlines a strategy to reveal macroscopic physical observables of TSS in compounds with a semimetallic bulk band structure, as is the case in multifunctional Heusler compounds, thereby opening up opportunities for their utilization in hybrid quantum structures.