Sweep and ejection events in turbulent boundary layer flows have been explored for half a century now to describe eddies impacting turbulent stresses. Yet, moving these studies from their current diagnostic phase to a prognostic form remains a formidable challenge. Here, a cumulant expansion is used to derive a link between the transport of shear stress and the balance of local sweep and ejection events. Cumulant expansion is further used to connect this transport to a metric of asymmetry in the streamwise velocity distribution. These relations are employed to develop two so-called structural models for predicting the turbulent stress transport, which is traditionally neglected in first-order closure of the shear stress budget. Several datasets collected in rough-wall conditions are used to show the importance of the transport term in the roughness sublayer and to demonstrate the predictive skill of the two structural models. The model parameters are invariant to the tested range of Reynolds number and surface roughness, indicating the structural similarity between the velocity asymmetry, sweep/ejection balance, and stress transport may be universal and independent of roughness. Finally, the implementation of the structural models for improved closure schemes of the shear stress budget in modeling applications and wall-modeling in large-eddy simulations are discussed.