A majority of breast cancers (BC) are age-related and we seek to determine what cellular and molecular changes occur in breast tissue with age that make women more susceptible to cancer initiation. Immune-epithelial cell interactions are important during mammary gland development and the immune system plays an important role in BC progression. The composition of human immune cell populations is known to change in peripheral blood with age and in breast tissue during BC progression. Less is known about changes in immune populations in normal breast tissue and how their interactions with mammary epithelia change with age. We quantified densities of T cells, B cells, and macrophage subsets in pathologically normal breast tissue from 122 different women who ranged in age from 24 to 74 years old. Donor-matched peripheral blood from a subset of 20 donors was analyzed by flow cytometry. Tissue immune cell densities and localizations relative to the epithelium were quantified in situ with machine learning-based image analyses of multiplex immunohistochemistry-stained tissue sections. In situ results were corroborated with flow cytometry analyses of peri-epithelial immune cells from primary breast tissue preparations and transcriptome analyses of public data from bulk tissue reduction mammoplasties. Proportions of immune cell subsets in breast tissue and donor-matched peripheral blood were not correlated. Density (cells/mm2) of T and B lymphocytes in situ decreased with age. T cells and macrophages preferentially localized near or within epithelial bilayers, rather than the intralobular stroma. M2 macrophage density was higher than M1 macrophage density and this difference was due to higher density of M2 in the intralobular stroma. Transcriptional signature analyses suggested age-dependent decline in adaptive immune cell populations and functions and increased innate immune cell activity. T cells and macrophages are so intimately associated with the epithelia that they are embedded within the bilayer, suggesting an important role for immune-epithelial cell interactions. Age-associated decreased T cell density in peri-epithelial regions, and increased M2 macrophage density in intralobular stroma suggests the emergence of a tissue microenvironment that is simultaneously immune-senescent and immunosuppressive with age.