Background: This study investigated the association of vitamin D3 levels with breast cancer risk and progression in African-Americans and Hispanics. Methods: A total of 237 African-American (Cases = 119, Control = 118) and 423 Hispanic women (Cases = 124, Control = 299) were recruited in the study. Blood samples were collected at the time of breast cancer screening and prior to cancer treatment for 4 weeks on average for the cases. The serum 25-hydroxyvitamin D (25(OH)D3) was measured at a Quest-Diagnostics facility. Results: The results showed that 69.2% of African-Americans and 37.8% of Hispanics had 25(OH)D3 levels below 20 ng/mL. The 25(OH)D3 level below 20 ng/mL was significantly associated with breast cancer in both African-Americans (OR = 2.5, 95% CI = 1.3-4.8) and Hispanics (OR = 1.9, 95% CI = 1.1-3.0). However, the predicted probabilities of breast cancer in African-Americans were significantly higher than in Hispanics (p < 0.001). The 25(OH)D3 below 20 ng/mL was significantly associated with triple negative breast cancer (TNBC) in African-Americans (OR = 5.4, p = 0.02, 95% CI = 1.4-15), but not in Hispanics in our cohort of participants. Levels of 25(OH)D3 below 26 ng/mL predicts a decrease in disease-free survival, but it was not an independent predictor. Conclusions: Our data shows an association between 25(OH)D3 levels and the risk of breast cancer. Further studies on the relationship between 25(OH)D3 level and breast cancer risk are warranted.