Antiferromagnetism (AFM) underlies some of the most fascinating and widely-studied questions in modern condensed matter physics. The work presented here applies the unique capabilities of polarization-resolved optical techniques to the study of phase transitions, magnetic ground states, and excitations in two AFM materials: alpha-RuCl_3 and Fe_(1/3)NbS_2. While the motivation for each project is quite different, these two compounds share a few common properties: both are layered materials where the magnetism can be treated as quasi-two dimensional, and both order antiferromagnetically at low temperature and zero applied field. Throughout this dissertation, we will explore how the interesting properties of both systems emerge from the physics of frustrated magnetism.

First, we consider the low-energy magnetic excitations in the Kitaev quantum spin liquid candidate material, RuCl_3. While this material orders at T$_N=$7 K in zero applied field, it can be driven into a magnetically disordered phase by the application of an in-plane magnetic field with a critical field H$_c \sim$ 7.5 T. In the ordered phase and in zero applied magnetic field, inelastic neutron scattering measurements observed peaks consistent with magnons, in addition to an unusual continuum of scattering centered at zero wave vector. This continuum was found to persist above H$_c$ and was interpreted as a possible signature of fractionalized excitations, begging the question of whether a field-induced spin liquid phase may exist in the vicinity of H$_c$.

Tracking the evolution of the magnetic excitations in RuCl_3 is thus of central importance to understanding the proximity of this system to a fractionalized spin liquid. Using time-domain terahertz (THz) spectroscopy, we quantitatively characterize the zero wave vector spectrum of this material in the phase space of temperature, applied field and THz polarization. Below H$_c$ we observe two sharp single-magnon absorption peaks and two broad multi-magnon peaks on top of a large continuum of absorption. This THz continuum does not depend on temperature or field. By way of spectral weight analysis we attribute this feature largely to electric dipole, rather than magnetic dipole absorption. We find that the spectral weight from magnetic absorption at low energies is largely accounted for by magnons, rather than fractional excitations, and the contribution from a potential magnetic continuum does not grow even approaching H$_c$. Furthermore, the high resolution of the THz spectra enables close comparison with theoretical models. Below the N eel temperature and critical field, we find that the essential features of our data are reproduced by calculations based on linear spin wave theory when a C$_3$-breaking distortion of the honeycomb lattice and the presence of structural domains are taken into account. We quantify the magnon selection rules, identify a low-field crossover transition and provide insight into the values of parameters in the magnetic Hamiltonian describing RuCl_3.

In the second part of this dissertation, we study the transition metal dichalcogenide (TMD) compound, Fe_(1/3)NbS_2 , where AFM order arises at T$_N$~45 K on a triangular Fe-superlattice intercalated between TMD layers. Notably, current-induced resistance switching and multi-stable memory effects were recently observed in this material when cooled into the magnetic phase. We develop a photo-thermally modulated scanning birefringence microscopy technique which enables the detection of rotational symmetry breaking as as function of temperature and spatial location on the sample. In Fe_(1/3)NbS_2 we detect a sudden onset of optical birefringence at T$_N$, indicating the lowering of $C_3$ rotational symmetry of the triangular lattice to at most $C_2$ in the AFM phase. Across the sample surface, we image three distinct orientations of this symmetry-breaking, corresponding to three magnetic-nematic domains which are locked to the crystallographic axes of the Fe- superlattice.

\par Employing the language of nematic order in crystalline materials developed in the context of the tetragonal Fe-based superconductors, we attribute our observations in Fe_(1/3)NbS_2 to a coincident magnetic-nematic phase transition at at the N eel temperature, albeit on the triangular lattice. Qualitatively, at T_N a distortion along one of the three directions of Fe-Fe bonds relieves the geometric frustration of Ising spins on a triangular lattice, allowing one of three degenerate ordered AFM phases to condense. A phenomenological Landau theory shows that nematicity in this compound is analogous to the three-state Potts model, and is hence described by a distinct Z_3 nematic order parameter. These ideas are then tested by showing that that the domain population, and hence the global nematic director, is continuously tunable by the application of uniaxial strain. This suggests that the anisotropy axes of response functions-- for instance the resistivity tensor-- can be continuously re-oriented by external perturbations such as strain or even current or magnetic field. In principle, this coupling to the nematic director could mediate the resistive switching behavior observed in Fe_(1/3)NbS_2 , indicating it may be of interest to search for AFM switching phenomena in the broad class of nematic-magnetic systems.