The performance of superconducting quantum circuits is primarily limited by dielectric loss due to interactions with two-level systems (TLSs). State-of-the-art circuits with engineered material interfaces are approaching a limit where dielectric loss from bulk substrates plays an important role. However, a microscopic understanding of dielectric loss in crystalline substrates is still lacking. In this work, we show that boron acceptors in silicon constitute a TLS bath that leads to an energy dissipation channel for superconducting circuits. We discuss how the electronic structure of boron acceptors leads to an effective TLS response in silicon. We sweep the boron concentration in silicon and demonstrate the bulk dielectric loss limit from boron acceptors. We show that boron-induced dielectric loss can be reduced in a magnetic field due to the spin-orbit structure of boron. This work provides the first detailed microscopic description of a TLS bath for superconducting circuits and demonstrates the need for ultrahigh-purity substrates for next-generation superconducting quantum processors. Published by the American Physical Society 2024