Intratumor heterogeneity associates with poor patient outcome. Stromal stiffening also accompanies cancer. Whether cancers demonstrate stiffness heterogeneity, and if this is linked to tumor cell heterogeneity remains unclear. We developed a method to measure the stiffness heterogeneity in human breast tumors that quantifies the stromal stiffness each cell experiences and permits visual registration with biomarkers of tumor progression. We present Spatially Transformed Inferential Force Map (STIFMap) which exploits computer vision to precisely automate atomic force microscopy (AFM) indentation combined with a trained convolutional neural network to predict stromal elasticity with micron-resolution using collagen morphological features and ground truth AFM data. We registered high-elasticity regions within human breast tumors colocalizing with markers of mechanical activation and an epithelial-to-mesenchymal transition (EMT). The findings highlight the utility of STIFMap to assess mechanical heterogeneity of human tumors across length scales from single cells to whole tissues and implicates stromal stiffness in tumor cell heterogeneity.