© 2016. The American Astronomical Society. All rights reserved. We analyze how passive galaxies at z ∼ 1.5 populate the mass-size plane as a function of their stellar age, to understand if the observed size growth with time can be explained with the appearance of larger quenched galaxies at lower redshift. We use a sample of 32 passive galaxies extracted from the Wide Field Camera 3 Infrared Spectroscopic Parallel (WISP) survey with spectroscopic redshift 1.3 ≲ z ≲ 2.05, specific star formation rates lower than 0.01 Gyr-1, and stellar masses above 4.5 × 1010 M Ȯ. All galaxies have spectrally determined stellar ages from fitting of their rest-frame optical spectra and photometry with stellar population models. When dividing our sample into young (age ≤2.1 Gyr) and old (age >2.1 Gyr) galaxies we do not find a significant trend in the distributions of the difference between the observed radius and that predicted by the mass-size relation. This result indicates that the relation between the galaxy age and its distance from the mass-size relation, if it exists, is rather shallow, with a slope α -0.6. At face value, this finding suggests that multiple dry and/or wet minor mergers, rather than the appearance of newly quenched galaxies, are mainly responsible for the observed time evolution of the mass-size relation in passive galaxies.