The presence of tumor-infiltrating lymphocytes (TIL) is a favorable prognostic factor in breast cancer, but what drives immune infiltration remains unknown. Here we examine if clonal heterogeneity, total mutation load, neoantigen load, copy number variations (CNV), gene- or pathway-level somatic mutations, or germline polymorphisms (SNP) are associated with immune metagene expression in breast cancer subtypes. Thirteen published immune metagenes correlated separately with genomic metrics in the 3 major breast cancer subtypes. We analyzed RNA-Seq, DNA copy number, mutation and germline SNP data of 627 ER positive, 207 HER2 positive, and 191 triple negative (TNBC) cancers from TCGA. P-values were adjusted for multiple comparisons, and permutation testing was used to assess false discovery rates. Increased immune metagene expression associated significantly with lower clonal heterogeneity estimated by MATH score in all subtypes and with a trend for lower overall mutation, neoantigen, and CNV loads in TNBC and HER2+ cancers. In ER+ cancers, mutation load, neoantigen load, and CNV load weakly but positively associated with immune infiltration, which reached significance for overall mutation load only. No highly recurrent single gene or pathway level mutations associated with immune infiltration. High immune gene expression and lower clonal heterogeneity in TNBC and HER2+ cancers suggest an immune pruning effect and equilibrium between immune surveillance and clonal expansion. Thus, immune checkpoint inhibitors may tip the balance in favor of immune surveillance in these cancers.