Tumors are characterized by extracellular matrix (ECM) deposition, remodeling, and cross-linking that drive fibrosis to stiffen the stroma and promote malignancy. The stiffened stroma enhances tumor cell growth, survival and migration and drives a mesenchymal transition. A stiff ECM also induces angiogenesis, hypoxia and compromises anti-tumor immunity. Not surprisingly, tumor aggression and poor patient prognosis correlate with degree of tissue fibrosis and level of stromal stiffness. In this review, we discuss the reciprocal interplay between tumor cells, cancer associated fibroblasts (CAF), immune cells and ECM stiffness in malignant transformation and cancer aggression. We discuss CAF heterogeneity and describe its impact on tumor development and aggression focusing on the role of CAFs in engineering the fibrotic tumor stroma and tuning tumor cell tension and modulating the immune response. To illustrate the role of mechanoreciprocity in tumor evolution we summarize data from breast cancer and pancreatic ductal carcinoma (PDAC) studies, and finish by discussing emerging anti-fibrotic strategies aimed at treating cancer.