The COVID-19 pandemic has had devastating direct consequences on the health of affected patients. It has also had a significant impact on the ability of unaffected children to be physically active. We evaluated the effect of deconditioning from social distancing and school shutdowns implemented during the COVID-19 pandemic on the cardiovascular fitness of healthy unaffected children. This is a single-center, retrospective case-control study performed in an urban tertiary referral center. A cohort of 10 healthy children that underwent cardiopulmonary exercise testing after COVID-19 hospital restrictions were lifted was compared to a matched cohort before COVID-19-related shutdowns on school and after-school activities. Comparisons of oxygen uptake (VO2) max and VO2 at anaerobic threshold between the pre- and post-COVID-19 cohorts were done. The VO2 max in the post-COVID cohort was significantly lower than in the pre-COVID cohort (39.1 vs. 44.7, p = 0.031). Only one out of ten patients had a higher VO2 max when compared to their matched pre-COVID control and was also the only patient with a documented history of participation in varsity-type athletics. The percentile of predicted VO2 was significantly lower in the post-COVID cohort (95% vs. 105%, p = 0.042). This study for the first time documented a significant measurable decline in physical fitness of healthy children as a result of the COVID-19 pandemic and its associated restrictions. Measures need to be identified that encourage and facilitate regular exercise in children in a way that are not solely dependent on school and organized after-school activities.