Topological materials often exhibit remarkably linear nonsaturating magnetoresistance (LMR), which is both of scientific and technological importance. However, the role of topologically nontrivial states in the emergence of such a behavior has eluded clear demonstration in experiments. Here, by reducing the coupling between the topological surface states (TSS) and the bulk carriers, we controllably tune the LMR behavior in Pt1-xAuxLuSb into distinct plateaus in Hall resistance, which we show arise from a quantum Hall phase. This allowed us to reveal how smearing of the Landau levels, which otherwise gives rise to a quantum Hall phase, results in an LMR behavior due to strong interaction between the TSS with a positive g factor and the bulk carriers. We establish that controlling the coupling strength between the surface and the bulk carriers in topological materials can bring about dramatic changes in their magnetotransport behavior. In addition, our work outlines a strategy to reveal macroscopic physical observables of TSS in compounds with a semimetallic bulk band structure, as is the case in multifunctional Heusler compounds, thereby opening up opportunities for their utilization in hybrid quantum structures.