In this thesis we study the representation theory of vertex operator algebras. The thesis consists of two parts. The first part deals with the connection among rationality, regularity and $C_2$-cofiniteness of vertex operator algebras. It is proved that if any $Z$-graded weak module for a vertex operator algebra V is completely reducible, then V is rational and $C_2$-cofinite. That is, V is regular. This gives a natural characterization of regular vertex operator algebras. Motivated by classification of rational vertex operator algebras with central charge c = 1. We compute the quantum dimensions of irreducible modules of the rational and C2-cofinite vertex operator algebra $V_{L_2}^{A_4}$ in the other part.This result will be used to determine the fusion rules for this algebra.

Let \(V\) be a simple vertex operator algebra, and \(G\) a finite automorphism group of \(V\) such that \(V^G\) is regular. The definition of entries in \(S\)-matrix on \(V^G\) is discussed, and then is extended. The set of \(V^G\)-modules can be considered as a unitary space. In this paper, we obtain some connections between \(V\)-modules and \(V^G\)-modules over that unitary space. As an application, we determine the fusion rules for irreducible \(V^G\)-modules which occur as submodules of irreducible \(V\)-modules by the fusion rules for irreducible \(V\)-modules and by the structure of \(G\).

The quantum dimensions of modules for vertex operator algebras are defined and their properties are discussed systematically. The quantum dimensions of the Heisenberg vertex operator algebra modules, the Virasoro vertex operator algebra modules and the lattice vertex operator algebra modules are computed. A criterion for simple current modules of a rational vertex operator algebra is given. The possible values of the quantum dimensions are obtained for rational vertex operator algebras. A full Galois theory for rational vertex operator algebras is established using the quantum dimensions.

In this thesis, we study the structure of 3C-algebra and 5A-algebra constructed byLam-Yamada-Yamauchi. We mainly use relevant braiding matrices to establish the uniqueness of the vertex operator algebra structure of these two algebras. Besides, we also give the fusion rules for these two algebras.

In this thesis, we compute the fusion rules among the irreducible modules of $V_L$ - the vertex operator algebra associated with a positive-definite even lattice $L$, and then use them to determine the irreducible decomposition of fusion products of irreducible $V_L$-modules. Specifically, we establish the following results: the fusion product of an untwisted $V_L$-module and another one of twisted type is a $V_L$-module of twisted type while the fusion product of two twisted $V_L$-modules is a sum of untwisted modules satisfying a certain relation.

Let $V$ be a vertex operator algebra, $T\in \N$ and $(M^k, Y_{M^k})$ for $k=1, 2, 3$ be a $g_k$-twisted module, where $g_k$ are commuting automorphisms of $V$ such that $g_k^T=1$ for $k=1, 2, 3$ and $g_3=g_1g_2$. Suppose $I(\cdot, z)$ is an intertwining operator of type $\left(\begin{array}{c}M^{3} \ M^{1} M^{2} \end{array}\right) $. We construct an $A_{g_1g_2}(V)$-$A_{g_2}(V)$-bimodule $A_{g_1g_2, g_2}(M^1)$ which determines the action of $M^1$ from the bottom level of $M^2$ to the bottom level of $M^3$.

We introduce the notion of space of correlation functions associated with three modules $M^1, M^2$, and $M^3$ over a vertex operator algebra $V$. By studying the relations between the space of correlation functions with the space of intertwining operators and the bi-modules over Zhu's algebra $A(V)$, we prove a generalized version of the fusion rules theorem for vertex operator algebras. We also give the analog of Rota-Baxter operators for vertex operator algebras as a generalization of the Rota-Baxter operators for Lie algebras. We find some particular types of sub-algebras of the lattice vertex operator algebra $V_L$ to give examples of such operators. Using a general version of Rota-Baxter operators of vertex operator algebra, we find a tensor form of the Yang-Baxter equations for vertex operator algebras that generalizes the classical Yang-Baxter equation for Lie algebras.

Let $V$ be a vertex operator superalgebra and $\sigma$ the order $2$ automorphism \linebreak associated with the superstructure of $V$. For a finite order automorphism $g$ with $o(g\sigma)=T'$, we follow~\cite{DONG-LI-MASON--AGNV} to construct a sequence of associative algebras $\AGNV$ for $n\in \frac{1}{T'}\Z_+$ such that $A_{g,n-\frac{1}{T'}}(V)$ is a quotient of $\AGNV$. There is a bijection between the irreducible $\AGNV$-modules which cannot factor through $A_{g,n-\frac{1}{T'}}(V)$ and the irreducible admissible $g$-twisted $V$-modules. These results are then\newline applied to $g$-rational vertex operator superalgebras. In this case it is shown that $V$ is $g$-rational if and only if all the $\AGNV$ are finite-dimensional. Taking $n=0$ we obtain the associative algebra $\AGV$ constructed in~\cite{DONG-ZHAO--AGV}. With $g=1$ we recover $A_n(V)$ as in~\cite{JIANG-JIANG}.