The Kitaev model, a honeycomb network of spins with bond-dependent anisotropic interactions, is a rare example of a system with a quantum spin liquid ground state. Although most Kitaev model candidate materials eventually order magnetically due to additional non-Kitaev interactions, their bond-dependent anisotropy manifests in unusual spin dynamics. Recent research suggests that bond-dependent anisotropy can stabilize exotic magnetic phases on the geometrically frustrated triangular lattice. Unfortunately, few materials have been identified with simultaneous geometric frustration and bond-dependent anisotropy. Here, we report a frustrated triangular lattice with bond-dependent anisotropy in the cobalt-based van der Waals antiferromagnet CoI2. Momentum and energy-resolved inelastic neutron scattering measurements show substantial magnon decay and level repulsion. A thorough examination of excitations in both the paramagnetic and magnetically ordered states demonstrates that the bond-dependent anisotropy is the origin of the spiral order and the magnon decay found in CoI2. Our results provide the basis for future studies of the interplay between Kitaev magnetism and geometric frustration.

Magnons and phonons are fundamental quasiparticles in a solid and can be coupled together to form a hybrid quasi-particle. However, detailed experimental studies on the underlying Hamiltonian of this particle are rare for actual materials. Moreover, the anharmonicity of such magnetoelastic excitations remains largely unexplored, although it is essential for a proper understanding of their diverse thermodynamic behaviour and intrinsic zero-temperature decay. Here we show that in non-collinear antiferromagnets, a strong magnon-phonon coupling can significantly enhance the anharmonicity, resulting in the creation of magnetoelastic excitations and their spontaneous decay. By measuring the spin waves over the full Brillouin zone and carrying out anharmonic spin wave calculations using a Hamiltonian with an explicit magnon-phonon coupling, we have identified a hybrid magnetoelastic mode in (Y,Lu)MnO₃ and quantified its decay rate and the exchange-striction coupling term required to produce it.

Novel effects induced by nonmagnetic impurities in frustrated magnets and quantum spin liquid represent a highly nontrivial and interesting problem. A theoretical proposal of extended modulated spin structures induced by doping of such magnets, distinct from the well-known skyrmions has attracted significant interest. Here, we demonstrate that nonmagnetic impurities can produce such extended spin structures in h-YMnO₃, a triangular antiferromagnet with noncollinear magnetic order. Using inelastic neutron scattering (INS), we measured the full dynamical structure factor in Al-doped h-YMnO₃ and confirmed the presence of magnon damping with a clear momentum dependence. Our theoretical calculations can reproduce the key features of the INS data, supporting the formation of the proposed spin textures. As such, our study provides the first experimental confirmation of the impurity-induced spin textures. It offers new insights and understanding of the impurity effects in a broad class of noncollinear magnetic systems.