Within the field of magnetism, magnetic frustration is a generous source of new phases of matter with unusual and exciting physical properties. Magnetic order is dictated by the interdependence of local electronic states and their real-space interactions decorating materials lattices. These local states can compete or work together to drive differing magnetic states of matter, ranging from the typical collinear (anti)ferromagnets to more exotic, highly-entangled ground states with emergent phenomena in small spin systems (S = 1/2). These correlated phases are predicted to arise in materials lattices that incite multiple equivalent interactions without a readily apparent magnetic ground state configuration in so called magnetic frustration. In turn, this frustration can lead to massively degenerate classical ground states hinting that instead a novel entangled phase could arise from the competition. Even with decades of research in this field, frustrated magnets remain an exciting and growing field of research as new magnetic phases arediscovered.

Herein, the physical properties of insulating lanthanide-based frustrated magnets areexplored from select members of the ALnX2 (A = alkali, Ln = lanthanide, X = chalcogenide) family of materials. This materials family crystallizes in various structures dictated by the ratio of the lanthanide radius to the alkali and chalcogenide radii, and this thesis focuses on two frustrated crystal lattices in this family, the equilateral triangular lattice and the elongated diamond lattice. We find that triangular lattice NaYbO2 does not conventionally order to 50 mK and instead contains an unconventional quantum disordered ground state that is tunable in an external magnetic field. However, structurally-similar triangular lattice KCeO2 magnetically orders below 300 mK with a small magnetic moment. Additionally, we show how the Heisenberg J1 - J2 model can be applied to the elongated magnetic diamond lattice in LiYbO2 and NaCeO2 where collinear or spiral magnetic order arises depending on the ratio of J2/J1. In LiYbO2, spiral magnetic order undergoes an incommensurate-to-commensurate structure under an external field below 1 K while NaCeO2 exhibits long range collinear antiferromagnetic order below 3.2 K. Overall, these materials studies advance the dependence of frustrated magnetic phases on external parameters, lanthanide character, and lattice geometry.

In strongly correlated electron systems, the independent electron approximation fails and electron-electron correlations must be taken into consideration. Such systems display an abundance of technologically useful behaviors including metal-insulator transitions, superconductivity, and colossal magnetoresistance. Within the broad class of correlated materials, Mott insulators are a canonical example. These compounds have repulsive Coulomb interactions between on-site electrons large enough to open an energy gap between two portions of a valence band, thereby turning what conventional band theory would predict to be a metal into a (Mott) insulator. Despite many years of investigation, Mott insulators remain an exciting area of materials research owing in part to their proximity to quantum critical points and spin liquid ground states.

Here we have studied thin films of the Mott insulating rare earth titanates RTiO3 (R = Gd, Sm), and heterostructures of these compounds with the band insulator SrTiO3. At the RTiO3/SrTiO3 interface, electrostatic doping creates a two-dimensional electron liquid (2DEL) that resides within the SrTiO3 layers near the interface. In the case of thin SrTiO3 quantum wells between magnetic RTiO3 barriers, we used polarized neutron reflectometry and muon spin rotation to show that there is a critical well thickness below which the 2DEL electrons exhibit magnetic correlations. This critical thickness was found to be independent of the sign of the magnetic exchange interactions in the neighboring RTiO3 barriers. A follow up study on thin GdTiO3 layers embedded within SrTiO3 revealed magnetic dead layers at the interface of the two materials, but bulk-like ferrimagnetism within the center of the thin GdTiO3 layers. The independence of magnetism from the notable structural distortions observed within the thin GdTiO3 layers highlights the weak coupling between the magnetic exchange interactions and electronic bandwidth in this material. Finally, element-specific resonant X-ray measurements were used to probe electronic symmetry breaking within SmTiO3 films and found evidence of in-plane orbital polarization at room temperature. Together, these studies add to the collective understanding of electron-electron correlations within Mott insulating thin films, particularly those in proximity to a high-density 2D electron system.

Competing magnetic interactions between magnetic ions in triangular or hexagonallattice systems can create exotic magnetic ground states of interest for quantum computing and high temperature superconductivity. These triangular/hexagonal structural motifs have been studied extensively within several material systems, but one material system that has remains underexplored is the phosphide family LnCd3P3 (Ln = Lanthanide). LnCd3P3 compounds possess well-separated Ln3+ ions set in a triangular lattice, which makes this system another avenue to study frustrated magnetism and the resulting exotic magnetic ground states. Previously, the properties of the LaCd3P3 and CeCd3P3 have been examined in detail, and the crystal structure of PrCd3P3 has been studied with X-Ray Diffraction. In this work, a new method for synthesizing high quality polycrystalline samples of these LnCd3P3 compounds is presented. This new method was used to synthesize polycrystalline samples of NdCd3P3, which is a new member of the LnCd3P3 material system. We then conducted resistivity, heat capacity, and magnetization measurements of NdCd3P3 samples, which revealed magnetic transitions at temperatures below 20 K. Next, a flux growth method for synthesizing single crystals of heavier LnCd3P3 compounds was developed. PrCd3P3 single crystals were then grown using this method and studied with magnetization and heat capacity measurements.

This thesis is concerned with the construction, commissioning and usage of a high-pressure, laser-based, floating-zone furnace for the purpose of synthesizing bulk single crystals of quantum magnets and studying them via neutron scattering. Bulk single crystals are some of the most valuable experimental platforms in condensed matter physics. This is particularly true for studies of quantum magnetism since it allows for the use of single-crystal neutron scattering, which is the premier technique for analyzing both the static and dynamic behaviors of magnetic materials in conjunction with resolution of underlying anisotropies. The floating-zone crystal growth method has been especially impactful in this regard, as it has the potential to create high-quality crystals with large volumes, an essential feature for compatibility with the flux limitations of neutron scattering. Accordingly, broadening the phase-space of materials accessible by this technique is a direct means of helping the field move forward. The furnace described in this thesis represents such an improvement. Its laser-based design allows for floating-zone crystal growth at record-setting gas pressures with simultaneous access to high temperatures via a well-defined heating zone. These features make it suitable for growth of highly volatile, metastable and low surface tension compounds, all of which are challenging for traditional floating zone-furnaces.

I begin with a detailed overview of the floating-zone growth technique, including its advantages, disadvantages and the manner in which it is conducted (Chapter 2). This sets the stage for a description of the furnace (Chapter 3), including its key design features and the conduction of a variety of successful commissioning growths to demonstrate its abilities. I also provide a brief reflection on challenges associated with high-pressure, laser-based, floating-zone crystal growth.

Following an attempt at defining quantum magnetism and an introduction of the basic principles of neutron scattering(Chapter 4) detailed studies of two of the commissioning materials (both quantum magnets) are presented. The first study (Chapter 5) elucidates the role of chemical disorder in modifying the fragmented magnetic state found in Nd2Zr2O7 through a combination of thermodynamic measurements, electron probe microanalysis and neutron scattering. It is demonstrated that growth under high-pressure allows for a notable reduction in chemical disorder, which modifies the temperature range over which the different spin correlations of the fragmented state are stabilized. The second study (Chapter 6) centers around a detailed survey of the magnetic excitations in a highly stoichiometric Li2CuO2 sample (enabled by the high-pressure growth environment) via inelastic neutron scattering. The resulting data allowed for refinement of the existing exchange model and also revealed the presence of multi-magnon bound states. This provides the first reported observation of multi-magnon bound sates in a ferromagnetic chain by neutron scattering.

The final study presented (Chapter 7) summarizes an attempt to grow single-crystal Eu2Ir2O7 via high-pressure floating-zone. Due to persistent decomposition at high temperatures, a detailed study on polycrystalline carrier-doped (Eu1−xCax )2Ir2O7 was carried out instead. Using a wide variety of experimental probes it is demonstrated that depression of the thermal metal-insulator transition by carrier doping leads to the formationof an electronically phase-separated state in these samples, without observable change in the lattice symmetry. Comparison to literature shows that formation of this phase-separated state is dependent on synthesis method, indicating that even small amounts of disorder can dramatically modify the properties.

Spin-orbit coupling forms the basis for a number of novel magnetic and electronic phenomena, and significant research activity is dedicated to the interplay between spin-orbit coupling and other fundamental atomic interactions. The fundamental understanding of the effects of spin-orbit coupling and the chemical tuning of spin-orbit coupling strength will be pivotal to the future implementation of a number of exotic magnetic and electronic properties in next-generation electronic devices. I have explored two classes of materials featuring varying scales of spin-orbit coupling -- lacunar spinels and ruthenium halide perovskites -- through the development of magnetic characterization techniques and atomic structure determination. The magnetic characterization methods demonstrated and developed in this thesis lay the groundwork for the future characterization of materials for applications in spintronics and quantum computing.

Lacunar spinels with the chemical formula AB₄Q₈ (A = Al, Ga, Ge; B = V, Mo, Nb, Ta; and Q = S, Se) feature a spinel-like structure with ordered vacancies on half of the spinel A-sites, resulting in tetrahedral clusters of B-ions across which an S = 1/2 moment is localized. Structural distortions acting on these clusters are pivotal to the magnetic and electronic properties of lacunar spinels. In 3d lacunar spinels GaV₄S₈ and GaV₄Se₈, a rhombohedral distortion enables the Dzyaloshinskii-Moriya interaction, a consequence of spin-orbit perturbations on symmetric exchange, resulting in nanoscaled magnetic textures such as cycloids and topologically protected magnetic quasiparticles known as skyrmions. I present detailed magnetic phase diagram determination for Néel-type skyrmion hosts GaV₄S₈ and GaV₄Se₈ through a combined magnetoentropic mapping and computational approach which demonstrates the utility of the technique in uniaxial skyrmion hosts. The measured phase diagrams additionally reveal that unknown magnetic phases at low temperature in GaV₄Se₈ are associated with anomalously high entropy signatures.

Lacunar spinels with stronger spin-orbit coupling such as GaNb₄S₈, GaNb₄Se₈, and GaTa₄Se₈ feature low temperature tetragonal distortions with a dimerized spin-singlet ground state at low temperatures and pressure-driven insulator-to-metal transitions (IMTs) and superconductivity. Recent attention due to the confirmation of a spin-orbit driven J_eff = 3/2 state in GaTa₄Se₈ has highlighted subtle differences in the structural distortions between GaNb₄S₈, GaNb₄Se₈, and GaTa₄Se₈. I present a powder and single-crystal structural determination of GaNb₄Se₈, which features an additional structural transition to the primitive cubic space group P4 ̅3m and the coexistence of cubic and tetragonal phases down to low temperatures.

Finally, Ru halides are a relatively unexplored phase space considering the recent attention on several exciting Ru containing compounds such as the Kitaev spin-liquid candidate α–RuCl₃. I will present the magnetic characterization of a variety of perovskite-derived Ru^III and Ru^IV halide compounds synthesized by Dr. Pratap Vishnoi. In vacancy-ordered double perovskites A₂RuX₆ (A = K, Cs; X = Cl, Br) are analyzed in the context of Kotani's theory for isolated single-ion like magnetic moments and demonstrate the variation of spin-orbit coupling strength through chemical tuning. Other perovskite-based hybrid Ru halides are characterized in the context of both Curie-Weiss and Kotani theories, highlighting the diversity of the structural motifs and magnetic properties in perovskite-derived Ru halides.

The discovery of $A$V$_3$Sb$_5$ ($A$ = K, Rb, Cs) compounds with a kagome net structure has unveiled their intriguing properties including superconductivity and unique topological characteristics in their electronic structure. These materials also exhibit charge density wave (CDW) ordering, which manifests as a distortion known as a breathing mode in the kagome layers. It has been proposed that this CDW ordering arises from nesting effects between saddle points on the Fermi surface.

To contribute to the evolving understanding of this fascinating class of materials, this thesis presents a comprehensive exploration of diverse kagome materials. The focus lies on conducting calculations that delve into the Fermi surface nesting and Lindhard susceptibility of CsV$_3$Sb$_5$, a prominent member of the $A$V$_3$Sb$_5$ family. Furthermore, the thesis thoroughly investigates the coupling between CDW and superconducting (SC) states through experimental and computational approaches, particularly by introducing hole doping into the systems. The resulting phase diagrams for $A$V$_3$Sb$_5$ unveil the profound impact of slight carrier doping on the SC and CDW orders in these materials. In addition, this thesis presents a comprehensive study of other kagome-based materials, such as the members of the \textit{A}\textit{M}$_3$\textit{X}$_4$ family, including a focused analysis of YbV$_3$Sb$_4$ and EuV$_3$Sb$_4$. The research also delves into the properties of the $R$V$_6$Sn$_6$ compounds, characterized by a vanadium kagome-based structure, and investigates the intriguing characteristics of the $RM_3Pn_3$ family featuring a triangular lattice, which shares similar signatures of instabilities.

Throughout the thesis, a spotlight is cast on various aspects, including Fermi surface nesting-driven instabilities, CDW phenomena, and magnetic behaviors within these materials. Meticulous experimental investigations and advanced theoretical analyses offer valuable insights into the complex interplay between electronic structures, CDW ordering, and superconductivity. The findings challenge previous assumptions and classifications, emphasizing the critical role of electron-phonon interactions and complex electronic correlations in shaping the observed behaviors of these materials. Overall, this dissertation contributes to a deeper understanding of the kagome materials and underscores the potential of these materials for realizing exotic electronic states and offers new avenues for exploring their unique physical phenomena.

Quantum spin liquids are an emergent state of matter, where the interplay betweendifferent exchange interactions can facilitate long range magnetic disorder. The disordered ground state of HKQSL has unique physical properties, of special interest is their potential to facilitate topologically protected qubits. We have successfully fabricated a candidate QSL material following the Heisenberg-Kitaev model: NaRuO2, using a rather unusual solid state reaction approach involving out of equilibrium synthesis and fine tuned parameters for inorganic reactions. NaRuO2 exists at a unique spot in the Na-Ru-O phase space, being able to support such a fragile ground state. Investigating the Na-Ru-O phase space using the aforementioned solid state reaction to further distinguish it from other compounds, notably the very similar Na2RuO3. The same fabrication technique was also used to create a dopant series where Fe ions replace some of the Ru ions in the lattice, making the same layered structure, albeit one with a different ground state: spin glass. Characterizing the ground state of this HKQSL candidate was done using various methods. The fine structure and spin orbit coupling were probed using synchrotron x-ray techniques: X ray absorption spectroscopy (XAS) and Resonant Inelastic X-ray Scattering (RIXS). The macro lattice properties: electric, thermal and magnetic, were measured at low temperatures and showed the inherent disorder of the system at the ground state. Finally, the nature of the low-temperature magnetic fluctuations was explored using inelastic neutron scattering (INS) and muon spin resonance (μSR), revealing continuum excitations that may be of quantum origins. Quantum critical scaling methods were used to analyze the INS spectrum, confirming the proximity of the NaRuO2 ground state to a quantum phase transition.

In the diverse catalog of `quantum materials' one of the most beguiling parameters is the spin-orbit coupling. This dissertation is concerned with how spin-orbit coupling effects exotic physics in correlated transition metal oxides. The materials described herein are all assembled from corner-sharing octahedra (Ir,Ru)O$_6$, and are hole-doped from the $d^5$ configuration, yet they contain a diversity of electronic and magnetic phases. A combination of resonant X-ray scattering and neutron scattering were employed to try to unravel the nature of the order and fluctuations in these materials.

First we examine the unusual Lifshitz transition in metallic Sr$_2$Ir$_x$Ru$_{1-x}$O$_4$, where we unveil a transition from itinerant to local moment behavior that might speak to quantum criticality. Second we explore the closely related compound Sr$_3$(Ir$_{1-x}$Ru$_x$)$_2$O$_7$, where the antiferromagnetism persists to remarkably high Ru content. Third is Sr$_3$Ir$_2$O$_7$F$_2$, which is like a fully hole-doped and distorted analog to Sr$_3$Ir$_2$O$_7$; there the spin-orbit excitons are thoroughly compared to theory. Last is (Ca$_x$Nd$_{1-x}$)$_2$Ir$_2$O$_7$, a tunable magnetic semimetal with potential for interesting topological states.