Heavy alcohol consumption is associated with deleterious changes in the brain but associations of moderate alcohol intake are not well understood. We examined the association of alcohol consumption with brain white matter health in 377 middle-aged men (56-66 years old; mean 61.8 ± 2.6 years) who were participants in the Vietnam Era Twin Study of Aging (VETSA). T1-, T2-, proton density-, and diffusion-weighted magnetic resonance images were obtained. Diffusion measures were quantified from 12 major white matter tracts. Global white matter lesion (WML) burden was also quantified. Mixed effects linear models examined differences in diffusivity and WMLs by amount of alcohol intake. Analyses adjusted for numerous demographic, health, and lifestyle variables. An inverted-U association was found between alcohol intake and fractional anisotropy (FA) in several tracts, including the inferior-frontal-occipital fasciculus, uncinate fasciculus, superior longitudinal fasciculus, the forceps minor and the anterior thalamic radiations. In these tracts, FA increased with increasing alcohol intake, peaking with moderate alcohol intake (9-28 drinks in 14 days), and declining with heavier intake. Associations remained significant after exclusion of individuals with diabetes or hypertension. There was a U-shaped association in WML burden with highest burden among never drinkers and heavy drinkers (>28 drinks in 14 days). This association was no longer significant after exclusion of individuals with hypertension, since WML burden among heavy drinkers no longer differed from that of other drinkers. This suggests that hypertension related to heavy alcohol intake may contribute to WML burden observed among heavy drinkers. Together, these correlational results suggest that among middle-aged men, moderate drinking may be associated with metrics of better white matter health, particularly microstructural measures, whereas drinking beyond recommended guidelines may be associated with both microstructural and macrostructural white matter damage.