Estradiol, normally considered a female hormone, appears to play a significant role in men in a variety of physiologic functions, such as bone metabolism, cardiovascular health, and testicular function. As such, estradiol has been targeted by male reproductive and sexual medicine specialists to help treat conditions such as infertility and hypogonadism. The compounds that modulate estradiol levels in these clinical conditions are referred to as selective estrogen receptor modulators (SERMs) and aromatase inhibitors (AIs). In a certain subset of infertile men, particularly those with hypogonadism, or those who have a low serum testosterone to estradiol ratio, there is some evidence suggesting that SERMs and AIs can reverse the low serum testosterone levels or the testosterone to estradiol imbalance and occasionally improve any associated infertile or subfertile state. This review focuses on the role these SERMs and AIs play in the aforementioned reproductive conditions.