The preference for media that confirms prior attitudes and beliefs is problematic in democratic societies based on dialogue and joint deliberation. Over the last decades, partisan selective exposure (PSE) is argued to have increased along with other indices of polarization. We address the question of the increase in PSE, and possible differences by party, ideology and ideological extremity. Using data from the Pew Research Survey, we analyzed self-reported media consumption in 8 nationally representative surveys from the period 2000–2012 (n = 23,381). We relied on previous research on ideological classification of media outlets to conduct confirmatory factor analyses establishing the existence of two different variables, conservative and liberal media consumption. We predicted latent variables of media consumption using Item Response Theory models and analyzed the trajectories running latent growth curve models. An unconditional growth model revealeda general and sustained increase in PSE across ideological groups over time. Republicans showed a greater increase over time than did Democrats, after controlling for demographics. Introducing ideological extremity in the model revealed no differences in the trajectories of PSE between liberals and extreme liberals, whereas subjects identified as “very conservative” show a much steeper increase in PSE than any other group, whereas conservatives showed the lowest growth over time. We discuss theoretical implications for ongoing debates about political polarization and ideological asymmetry.