Topological phases of matter are purely quantum mechanical and have no classical analogue. Most phases in nature can be classified and studied classically through the concept of symmetry breaking and its theoretical description, Landau-Ginzburg field theory. In contrast to the general wisdom of Landau-Ginzburg field theory, the topological phases share the same symmetry as a trivial insulator and still are different phases. Having gapped spectrum in bulk, they support a metallic edge excition robust against symmetry-respecting perturbation or an emergent fractional excitation in bulk. Following after fractional quantum Hall fluids, many topologically orderd phases, such as spin liquids and topological insulators, have been found and studied. Spin liquids are disordered phases of frustrated antiferromagnets and do not freeze and order even at the lowest temperature. They support an electrically neutral spin-1/2 excitation, which does not exist in a microscopic scale, with emergent dynamical gauge field in bulk and do not have an adiabatic path to a trivial paramagnet phase. The topological insulators are time-reversal symmetric band insulators which cannot evolve smoothly into a trivial insulator with the symmetry, and they have been intensely studied theoretically and experimetally for the last decade. Though the topological insulators are inherently non-interacting systems, they have exotic gapless edge and surface states which demonstrate many interesting quantum phenomena such as fractionalization, axionic electromagnetism, and half quantum Hall effect.

In this thesis, we study various quantum phenomena of the topological phases, mainly of the topological insulator and its close relatives in which the physics of spin liquids has been merged into. As they are intrinsically quantum many-body states, the quantum field theory is an invaluable tool to explore the venue of the phases and will be used thorougly in this work.

First, we present a topological field theoretic description of the topological insulators and translational symmetric Z2 spin liquid. We demostrate that abelian BF theory is an effective field theory for the two and three dimensional topological insulators. We show that the phenomenologies of BF theory are consistent with those of the topological insulators such as gapless edge and surface excitations and response functions to external gauge field. The same form of the BF theory is also applicable to study the certain classes of Z2 spin liquids with the gapless edge states. We closely examine the possible gapless edge theory of the BF theory and the translational symmetric spin liquids, and we show that the effective BF theories capture the bulk properties as well as the gapless edge spectrum of the spin liquid. Also we discuss the zero-temperature phase diagram of a thin film three dimensional topological insulators with two competing mass terms: time-reversal symmetric exciton condensation and time-reversal symmetry breaking Zeeman effect. There are two phases, quantum spin Hall phase and quantum anomalous Hall phase, and both phases support fractional excitations. We derive an effective topological field theory for the fractional excitations and examine the origin of the fractional excitations.

Next we consider the relatives of the topological insulators. First we discuss the proximate symmetry broken phases of topological Mott insulator, a U(1) spin liquid with fermionic spinons in the topological insulator state. The topologically non-trivial band structure of the spinon generates the axion term for the gauge field, and we show that the axion term changes the nature of the confined phase of the spin liquid. Contrast to the conventional confined phase which has only a bond order, the confined phase of the spin liquid is in general a coexisting phase of the two different orders : a current order and a bond order. We then consider another relative of the topological insulator: a bosonic symmetry protected topological phase in two dimensional space, with both PSU(N) and time reversal symmetry. We develop an effective field theory for the phase, which is a SU(N) principal chiral model with a theta-term, and reveal the physics of the topological phase in detail.

Finally we consider a topological semimetal, namely a Weyl semimetal. We first demonstrate that the Weyl semimetal can be realized in the magnetically doped topological band insulators. We explicitly derive the low energy theory of Weyl points from the general continuum Hamiltonian and tight-binding model of the topological insulators. Then we study superconducting instabilities of doped Weyl semimetals. We consider a minimal model for a Weyl semimetal and study the superconducting instabilities induced by a short-ranged attractive interaction. With the interaction, we find two competing states: a fully gapped finite-momentum pairing state and a nodal even-parity pairing state. We show that, in a mean field approximateion, the finite-momentum pairing state wins energetically against the usual even-parity paired state. We also show that exotic modes are localized at the full and half quantum vortices of the finite-momentum pairing state.