The factors contributing to the rapid emergence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) BA.4 and BA.5 subvariants in populations that experienced recent surges of BA.2 and BA.2.12.1 infections are not understood. Neutralizing antibodies (NAbs) are likely to protect against severe disease if present in sufficient quantity. We found that after BA.2 or BA.2.12.1 infection, NAb responses were largely cross-neutralizing but were much less effective against BA.5. In addition, individuals who were infected and treated early with nirmatrelvir/ritonavir (Paxlovid) had lower NAb levels than untreated individuals.