Understanding the contribution of the host's genetic background to cancer immunity may lead to improved stratification for immunotherapy and to the identification of novel therapeutic targets. We investigated the effect of common and rare germline variants on 139 well-defined immune traits in ∼9000 cancer patients enrolled in TCGA. High heritability was observed for estimates of NK cell and T cell subset infiltration and for interferon signaling. Common variants of IFIH1, TMEM173 (STING1), and TMEM108 were associated with differential interferon signaling and variants mapping to RBL1 correlated with T cell subset abundance. Pathogenic or likely pathogenic variants in BRCA1 and in genes involved in telomere stabilization and Wnt-β-catenin also acted as immune modulators. Our findings provide evidence for the impact of germline genetics on the composition and functional orientation of the tumor immune microenvironment. The curated datasets, variants, and genes identified provide a resource toward further understanding of tumor-immune interactions.