Proteasome inhibitors are widely used as therapeutics and research tools, and typically target one of the three active sites, each present twice in the proteasome complex. An endogeneous proteasome inhibitor, PI31, was identified 30 years ago, but its inhibitory mechanism has remained unclear. Here, we identify the mechanism of Saccharomyces cerevisiae PI31, also known as Fub1. Using cryo-electron microscopy (cryo-EM), we show that the conserved carboxy-terminal domain of Fub1 is present inside the proteasome's barrel-shaped core particle (CP), where it simultaneously interacts with all six active sites. Targeted mutations of Fub1 disrupt proteasome inhibition at one active site, while leaving the other sites unaffected. Fub1 itself evades degradation through distinct mechanisms at each active site. The gate that allows substrates to access the CP is constitutively closed, and Fub1 is enriched in mutant CPs with an abnormally open gate, suggesting that Fub1 may function to neutralize aberrant proteasomes, thereby ensuring the fidelity of proteasome-mediated protein degradation.