Organic-inorganic layered perovskites, or Ruddlesden-Popper perovskites, are two-dimensional quantum wells with layers of lead-halide octahedra stacked between organic ligand barriers. The combination of their dielectric confinement and ionic sublattice results in excitonic excitations with substantial binding energies that are strongly coupled to the surrounding soft, polar lattice. However, the ligand environment in layered perovskites can significantly alter their optical properties due to the complex dynamic disorder of the soft perovskite lattice. Here, we infer dynamic disorder through phonon dephasing lifetimes initiated by resonant impulsive stimulated Raman photoexcitation followed by transient absorption probing for a variety of ligand substitutions. We demonstrate that vibrational relaxation in layered perovskite formed from flexible alkyl-amines as organic barriers is fast and relatively independent of the lattice temperature. Relaxation in layered perovskites spaced by aromatic amines is slower, although still fast relative to bulk inorganic lead bromide lattices, with a rate that is temperature dependent. Using molecular dynamics simulations, we explain the fast rates of relaxation by quantifying the large anharmonic coupling of the optical modes with the ligand layers and rationalize the temperature independence due to their amorphous packing. This work provides a molecular and time-domain depiction of the relaxation of nascent optical excitations and opens opportunities to understand how they couple to the complex layered perovskite lattice, elucidating design principles for optoelectronic devices.