Nondestructive ultrasound-based methods have been applied to evaluate the elastic properties of concrete materials. Although the wave modulus of elasticity of concrete frequently is reported to be higher than the static counterpart, the microstructural and physical mechanisms are not well understood. This study conducted a computational micromechanics to investigate the effects of aggregates and voids on both the effective wave modulus of elasticity and the static modulus of elasticity, based on concrete microstructures resolved with X-ray microtomography. It was demonstrated that the existence of void defects plays a significant role in the elastic properties of concrete compared with the aggregates. It was shown that the wave modulus of elasticity of concrete is higher than the static modulus of elasticity because of the existence of crack-like voids with small aspect ratios.