The purpose of this study was to determine the dynamic contributions of different immune cell subsets to primary and abscopal tumor regression after hypofractionated radiation therapy (hRT) and the impact of anti-PD-1 therapy. A bilateral syngeneic FSA1 fibrosarcoma model was used in immunocompetent C3H mice, with delayed inoculation to mimic primary and microscopic disease. The effect of tumor burden on intratumoral and splenic immune cell content was delineated as a prelude to hRT on macroscopic T1 tumors with 3 fractions of 8 Gy while microscopic T2 tumors were left untreated. This was performed with and without systemic anti-PD-1. Immune profiles within T1 and T2 tumors and in spleen changed drastically with tumor burden in untreated mice with infiltrating CD4+ content declining, while the proportion of CD4+ Tregs rose. Myeloid cell representation escalated in larger tumors, resulting in major decreases in the lymphoid:myeloid ratios. In general, activation of Tregs and myeloid-derived suppressor cells allow immunogenic tumors to grow, although their relative contributions change with time. The evidence suggests that primary T1 tumors self-regulate their immune content depending on their size and this can influence the lymphoid compartment of T2 tumors, especially with respect to Tregs. Tumor burden is a major confounding factor in immune analysis that has to be taken into consideration in experimental models and in the clinic. hRT caused complete local regression of primary tumors, which was accompanied by heavy infiltration of CD8+ T cells activated to express IFN-γ and PD-1; while certain myeloid populations diminished. In spite of this active infiltrate, primary hRT failed to generate the systemic conditions required to cause abscopal regression of unirradiated microscopic tumors unless PD-1 blockade, which on its own was ineffective, was added to the RT regimen. The combination further increased local and systemically activated CD8+ T cells, but few other changes. This study emphasizes the subtle interplay between the immune system and tumors as they grow and how difficult it is for local RT, which can generate a local immune response that may help with primary tumor regression, to overcome the systemic barriers that are generated so as to effect immune regression of even small abscopal lesions.