A key question in SARS-CoV-2 infection is why viral loads and patient outcomes vary dramatically across individuals. Because spatial-temporal dynamics of viral spread and immune response are challenging to study in vivo, we developed Spatial Immune Model of Coronavirus (SIMCoV), a scalable computational model that simulates hundreds of millions of lung cells, including respiratory epithelial cells and T cells. SIMCoV replicates viral growth dynamics observed in patients and shows how spatially dispersed infections can lead to increased viral loads. The model also shows how the timing and strength of the T cell response can affect viral persistence, oscillations, and control. By incorporating spatial interactions, SIMCoV provides a parsimonious explanation for the dramatically different viral load trajectories among patients by varying only the number of initial sites of infection and the magnitude and timing of the T cell immune response. When the branching airway structure of the lung is explicitly represented, we find that virus spreads faster than in a 2D layer of epithelial cells, but much more slowly than in an undifferentiated 3D grid or in a well-mixed differential equation model. These results illustrate how realistic, spatially explicit computational models can improve understanding of within-host dynamics of SARS-CoV-2 infection.