The collective behavior of lipids with diverse chemical and physical features determines a membrane's thermodynamic properties. Yet, the influence of lipid physicochemical properties on lipid dynamics, in particular interbilayer transport, remains underexplored. Here, we systematically investigate how the activation free energy of passive lipid transport depends on lipid chemistry and membrane phase. Through all-atom molecular dynamics simulations of 11 chemically distinct glycerophospholipids, we determine how lipid acyl chain length, unsaturation, and headgroup influence the free energy barriers for two elementary steps of lipid transport: lipid desorption, which is rate limiting, and lipid insertion into a membrane. Consistent with previous experimental measurements, we find that lipids with longer, saturated acyl chains have increased activation free energies compared to lipids with shorter, unsaturated chains. Lipids with different headgroups exhibit a range of activation free energies; however, no clear trend based solely on chemical structure can be identified, mirroring difficulties in the interpretation of previous experimental results. Compared to liquid-crystalline phase membranes, gel phase membranes exhibit substantially increased free energy barriers. Overall, we find that the activation free energy depends on a lipid's local hydrophobic environment in a membrane and that the free energy barrier for lipid insertion depends on a membrane's interfacial hydrophobicity. Both of these properties can be altered through changes in lipid acyl chain length, lipid headgroup, and membrane phase. Thus, the rate of lipid transport can be tuned through subtle changes in local membrane composition and order, suggesting an unappreciated role for nanoscale membrane domains in regulating cellular lipid dynamics.