Magnitude comparison tasks are used to assess the precision of numerical representations. Recent research, how-ever, questions the validity of different measures of magnitude comparison. We investigated the validity of five performancemeasures: overall RT, overall accuracy, numerical ratio effect (RT), numerical ratio effect (accuracy), and Weber fraction.Kindergarten and university students completed symbolic and non-symbolic magnitude comparison tasks and a math skill mea-sure. For children and adults, we calculated Chronbach’s α separately for each presentation format. All values were in theunacceptable range, indicating that the different indices were not measuring the same construct. For children, a multiple re-gression predicting KeyMath scores from symbolic and non-symbolic indices showed that only non-symbolic overall accuracyand symbolic overall RT were predictors. For adults, a multiple regression predicting French Kit scores showed that only thesymbolic numerical ratio effect (RT) was a predictor. No index demonstrated predictive validity across formats or age groups.