Cancer immunotherapy has revolutionized cancer treatment, and it relies heavily on the comprehensive understanding of the immune landscape of the tumor microenvironment (TME). Here, we obtain a detailed immune cell atlas of esophageal squamous cell carcinoma (ESCC) at single-cell resolution. Exhausted T and NK cells, regulatory T cells (Tregs), alternatively activated macrophages and tolerogenic dendritic cells are dominant in the TME. Transcriptional profiling coupled with T cell receptor (TCR) sequencing reveal lineage connections in T cell populations. CD8 T cells show continuous progression from pre-exhausted to exhausted T cells. While exhausted CD4, CD8 T and NK cells are major proliferative cell components in the TME, the crosstalk between macrophages and Tregs contributes to potential immunosuppression in the TME. Our results indicate several immunosuppressive mechanisms that may be simultaneously responsible for the failure of immuno-surveillance. Specific targeting of these immunosuppressive pathways may reactivate anti-tumor immune responses in ESCC.