The introduction of novel perennial grains into annual row crop rotations is proposed to increase soil ecosystem services and enhance plant-soil-microbial linkages because perennials provide deeper root systems and more continuous ground cover than annuals. While soil microbial communities underpin many ecosystem services, we know little about how soil microbial composition and diversity, and soil carbon storage, differ between soils of annual vs. perennial grain crops. We measured soil fungi: bacteria (F/B) ratios and soil carbon within the novel perennial intermediate wheatgrass (IWG; trademarked Kernza®) and tilled annual wheat and compared soil microbial diversity and community composition within their rhizosphere, shallow bulk soil (0–15 cm) and total bulk soil (0–90 cm). After three years, soil depth explained 30–40% and 12–22% of the variance in bacterial and fungal community composition, respectively, while crop type explained 10% and 9–16% of the variance, respectively. Fungal communities were most impacted by crop type in the rhizosphere and shallow bulk soil and less sensitive to differences in soil depth. In contrast, crop type had a smaller effect on bacterial communities which were more influenced by soil depth. IWG trended higher in soil carbon mass at 0–30 cm (p = 0.22) and had a higher (F/B) ratio than tilled annual wheat at depths below 15 cm, but tilled annual wheat had higher soil carbon concentration (p = 0.12) and soil carbon mass (p = 0.09) at the 60–90 cm soil depth. Our results indicate that fungi were more responsive than bacterial communities to crop type and that IWG has a higher fungal biomass and different fungal community composition than annual wheat at depth. However, despite these distinct differences in fungal communities in IWG compared to annual wheat, the differences did not translate into greater soil carbon mass in IWG at depth.