The endosomal trafficking of signaling membrane proteins, such as receptors, transporters and channels, is mediated by the retromer-mediated sorting machinery, composed of a cargo-selective vacuolar protein sorting trimer and a membrane-deforming subunit of sorting nexin proteins. Recent studies have shown that the isoforms, sorting nexin 5 (SNX5) and SNX6, have played distinctive regulatory roles in retrograde membrane trafficking. However, the molecular insight determined functional differences within the proteins remains unclear. We reported that SNX5 and SNX6 had distinct binding affinity to the cargo protein vesicular monoamine transporter 2 (VMAT2). SNX5, but not SNX6, specifically interacted with VMAT2 through the Phox domain, which contains an alpha-helix binding motif. Using chimeric mutagenesis, we identified that several key residues within this domain were unique in SNX5, but not SNX6, and played an auxiliary role in its binding to VMAT2. Importantly, we generated a set of mutant SNX6, in which the corresponding key residues were mutated to those in SNX5. In addition to the gain in binding affinity to VMAT2, their overexpression functionally rescued the altered retrograde trafficking of VMAT2 induced by siRNA-mediated depletion of SNX5. These data strongly suggest that SNX5 and SNX6 have different functions in retrograde membrane trafficking, which is determined by the different structural elements within the Phox domain of two proteins. Our work provides a new information on the role of SNX5 and SNX6 in the molecular regulation of retrograde membrane trafficking and vesicular membrane targeting in monoamine neurotransmission and neurological diseases.