We study one dimensional models of diatomic molecules where both the electrons and nuclei are treated as quantum particles, going beyond the usual Born-Oppenheimer approximation. The continuous system is approximated by a grid which computationally resembles a ladder, with the electrons living on one leg and the nuclei on the other. To simulate DMRG efficiently with this system, a three-site algorithm has been implemented. We also use a compression method to treat the long-range interactions between charged particles. We find that 1D diatomic molecules with spin-1/2 nuclei in the spin-triplet state will unbind when the mass of the nuclei reduces to only a few times larger than the electron mass, while the molecule with nuclei in the singlet state always binds, given the two electrons in their singlet state in both cases. We propose an improved scheme to do the time dependent variational principle (TDVP) in finite matrix product states (MPS) for two-dimensional systems or one-dimensional systems with long range interactions. We present a method to represent the time-evolving state in a MPS with its basis enriched by state-averaging with global Krylov vectors. We show that the projection error is significantly reduced so that precise time evolution can still be obtained even if a larger time step is used. Combined with the one-site TDVP, our approach provides a way to dynamically increase the bond dimension while still preserving unitarity for real time evolution. Our method can be more accurate and exhibit slower bond dimension growth than the conventional two-site TDVP. We apply our improved TDVP method to investigate the spin squeezing dynamics of the XXZ model of $1/r^\alpha$ interaction in two dimension. Comparing with the spin squeezing parameter and other observables obtained from discrete truncated Wigner approximation (DTWA), we verify the validity of DTWA and unveil the potential for this method to study dynamics of large-scale spin systems. Our results confirm the existence of large collective regime when $\alpha > 2$, which can be a guide for future experimental realizations. Combined with the purification method, our improved TDVP method is proved to be also useful to study the thermalization of the long-range interacting system.

The main content of this dissertation consists of two projects that I studied during my graduate career. Among them, the first project has more successful results and thus will be the major content of this dissertation. Besides discussing the details of each project, this dissertation also introduces the background knowledge and fundamental techniques of this area.

The first project is "recursion methods for strongly correlated quantum system}, which is also the main project". In this project, we present a method for extrapolation of real-time dynamical correlation functions which can improve the capability of matrix product state methods to compute spectral functions. Unlike the widely used linear prediction method, which ignores the origin of the data being extrapolated, our recursion methods utilize a representation of the wavefunction in terms of an expansion of the same wavefunction and its translations at earlier times. This recursion method is exact for noninteracting Fermi system. Surprisingly, the recursion method is also more robust than linear prediction at large interaction strength. We test this method on the Hubbard two-leg ladder, and present more accurate results for the spectral function than previous studies.

The second project is "Mapping the Hubbard model to the t-J model using ground state unitary transformations". The effective low-energy models of the Hubbard model are usually derived from perturbation theory. Here we derive the effective model of the Hubbard model in spin space and t-J space using a unitary transformation from numerical optimization. We represent the Hamiltonian as Matrix product state(MPO) and represent the unitary transformation using gates according to tensor network methods. We obtain this unitary transformation by optimizing the unitary transformation between the ground state of the Hubbard model and the projection of the Hubbard model ground state into spin space and t-J space. The unitary transformation we get from numerical optimization yields effective models that are in line with perturbation theories. This numerical optimization method starting from ground state provides another approach to analyze effective low-energy models of strongly correlated electron systems.

The density-matrix renormalization group (DMRG) invented by Steven R. White is a variational algorithm to search for the ground states of quantum many-body systems. Using the entanglement entropy as its organizing principle, DMRG stands as one of the most powerful methods in simulating two-dimensional (2D) quantum systems, and is especially useful for investigating strongly correlated systems that are otherwise challenging for analytical approaches. This thesis presents the applications and developments of DMRG and related tensor network methods in studying a variety of 2D doped and frustrated systems as well as their model reductions. Chapter 1 lays out the fundamentals of DMRG and tensor network states, along with multiple techniques for studying 2D systems. Chapter 2 presents our DMRG studies of the ground state phase diagram of the extended $t$-$J$ models. We found that while the models are consistent with the cuprates in antiferromagnetism and charge order, superconductivity nevertheless appears absent or marginal in hole-doped systems. Motivated by this discrepancy between the models and the cuprates, in Chapter 3 we carried out a DMRG-based downfolding of the parental three-band Hubbard model, seeking possible fixes to the previously studied single-band models. An effective model was derived via Wannier construction, which includes novel density-assisted hopping terms that appear to be important in enhancing hole-doped superconductivity. In Chapter 4, we examined the quantum spin nematic phase in the paradigmatic $S=1/2$ square-lattice $J_1$-$J_2$ ferro-antiferromagnetic Heisenberg model, employing a combination of DMRG and analytics. Our findings revealed that many-body effects induce significant contraction of the nematic phase compared to the na\"{i}ve expectation. Chapter 5 presents a study of the anisotropic $J_1^\Delta$-$J_3$ model on the honeycomb lattice, which is believed to be the fundamental model for many Kitaev material candidates upon adding bond-dependent terms. This chapter also includes my contribution to a study of the generalized Kitaev-$J_3$ model for $\alpha$-RuCl$_3$.