It is well-known that internal wave breaking is the principle cause of ocean mixing. The latter plays a role in shaping global climate and ocean ecology. To understand how internal wave loses instability and possibly breaks, in this thesis I present some new findings on the instability mechanisms of internal gravity waves due to wave-wave/topography interactions.

Parametric Subharmonic Instability (PSI) is one of the most important mechanisms that transfer energy from tidally-generated long internal waves to short steep waves. Breaking of these short waves results in diapycnal mixing through which oceanic abyssal stratification is maintained. It has long been believed that PSI is strongest between a primary internal wave and perturbative waves of half the frequency of the primary wave. Here, I rigorously show that this is not the case. Specifically, I show that neither the initial growth rate nor the maximum long-term amplification occur at the half frequency, and demonstrate that the dominant subharmonic waves have much longer wavelengths than previously thought.

Next I show that there exist internal gravity waves that are inherently unstable, that is, they cannot exist in nature for a long time. The instability mechanism is a one-way (irreversible) harmonic-generation resonance that permanently transfers the energy of an internal wave to its higher harmonics. I show that, in fact, there are a countably infinite number of such unstable waves. For the harmonic-generation resonance to take place, the nonlinear terms in the free surface boundary condition play a pivotal role, and the instability does not occur in a linearly-stratified fluid if a simplified boundary condition such as a rigid lid or a linearized boundary condition is employed. Harmonic-generation resonance presented here provides a mechanism for the transfer of internal wave energy to the higher-frequency part of the spectrum hence affecting, potentially significantly, the evolution of the internal waves spectrum.

Following the work on harmonic generations of internal waves, we show that monochromatic internal gravity waves in a stratified ocean can continuously descend to smaller scales of higher wave numbers after being reflected from no-penetration boundaries, provided their second harmonics satisfy dispersion relation. The wave with wave number $k$ and frequency $\omega$ first generate its second harmonic due to the non-linearity in the momentum equation and the free surface boundary. Then the resonant interaction of the parent wave with its second harmonic gives rise to a $(3k, \omega)$ wave upon reflection, starting from which a series of triad resonances continuously form in the vicinity of the frequency $\omega$ or $2\omega$. The energy of the internal wave is sent to waves with higher and higher wave numbers. The mechanism potentially explains how part of internal wave energy converts from large scales to that of turbulent mixing near reflecting boundaries, such as ocean ridges and steep continental slopes.

Internal waves mainly arise from wind disturbing the upper ocean mixed layer and barotropic tide flowing over topographic features in the ocean. I report a mechanism for conversion of tidal energy to internal gravity waves, through baratropic tidal currents interacting with ambient internal waves. The newly-generated internal waves have the same wave length as the ambient waves and frequencies higher by the tidal frequency. I show that they grow exponentially in time and continuously extract energy from the tidal currents. The mechanism exists in any stratified ocean with smooth density profiles and is only possible if a nonlinear free surface is taken into account. I show that for a typical water depth of 4000 m and density stratification 5\% in the Pacific Ocean, the energy flux from baratropic tide to plane internal waves via this mechanism can reach order of 1MWm$^{-1}$. It therefore can be significant enough to account for a fraction of the 1TW total tidal energy loss in the open deep ocean.

Ocean mixing has been seen increased where there is rough bottom topography in the ocean. I show in theory that, in a two dimensional domain, with a free surface taken into account, the interaction between an internal wave and corrugated bottom topography can efficiently generate a series of co-directional internal waves with higher wave numbers. As a result, the internal wave energy can be transferred to smaller scales where internal wave is more prone to breaking and being dissipated.