The COVID-19 pandemic has had a significant impact on public transit ridership in the United States, especially for rail transit. Land use, development density, and the pedestrian environment are strongly associated with station-level transit ridership. This study examines how these characteristics affect transit ridership pre- and post-COVID and how they differ across station types based on longitudinal data for 242 rail stations belonging to Bay Area Rapid Transit, San Diego Metropolitan Transit System, Sacramento Regional Transit, and LA Metro between 2019 and 2021. We found overall a 72% decrease in station-level ridership, but changes were not uniform. Station areas with a higher number of low-income workers and more retail or entertainment jobs tend to have lower ridership declines, while areas with a large number of high-income workers, high-wage jobs, and higher job accessibility by transit had more ridership losses. When comparing station area ridership and activity changes based on mobile phone user data, ridership declined more drastically than activity across all four rail systems, which implies that rail transit riders switched to other modes of transportation when accessing the station areas. Given these findings, it is likely that rail transit services oriented toward commute travel, especially core station areas with jobs for higher income workers, will continue to have an uneven recovery, posing critical implications for transit resilience planning and equity in the post-pandemic era. Considering sources of funding other than passenger fares to sustain rail transit, strategizing to reinvent and reinforce downtowns as destinations, and shifting rail transit services to appeal to non-commute travel can be promising strategies to support rail transit.