UC Irvine

Magnetization measurements were performed on CeCoIn 5 at temperatures down to 20 mK and magnetic fields up to 17 T applied along different crystallographic orientations. For field configurations nearly parallel to the ab plane (θ 40 and T≤50mK), we have found an intriguing vortex dynamics regime revealed by a hysteretic and metastable anomalous peak effect (APE), which gives evidence of surface barrier effects enhanced by antiferromagnetic fluctuations in the mixed state of CeCoIn 5. Furthermore, we have observed crossover features in the torque and magnetization traces at fields below H c2, which are consistent with vortices lattice phase transitions and with the anomalies speculated to be the Fulde-Ferrel-Larkin-Ovchinnikov (FFLO) superconducting state in CeCoIn 5. All of the above features were found to be dramatically perturbed in Ce 0.98Gd 0.02CoIn 5. © 2012 American Physical Society.