When deformed beyond their elastic limits, crystalline solids flow plastically via particle rearrangements localized around structural defects. Disordered solids also flow, but without obvious structural defects. We link structure to plasticity in disordered solids via a microscopic structural quantity, "softness," designed by machine learning to be maximally predictive of rearrangements. Experimental results and computations enabled us to measure the spatial correlations and strain response of softness, as well as two measures of plasticity: the size of rearrangements and the yield strain. All four quantities maintained remarkable commonality in their values for disordered packings of objects ranging from atoms to grains, spanning seven orders of magnitude in diameter and 13 orders of magnitude in elastic modulus. These commonalities link the spatial correlations and strain response of softness to rearrangement size and yield strain, respectively.