Spherical tokamak plasmas are typically overdense and thus inaccessible to externally-injected microwaves in the electron cyclotron range. The electrostatic electron Bernstein wave (EBW), however, provides a method to access the plasma core for heating and diagnostic purposes. Understanding the details of the coupling process to electromagnetic waves is thus important both for the interpretation of microwave diagnostic data and for assessing the feasibility of EBW heating and current drive. While the coupling is reasonably well-understood in the linear regime, nonlinear physics arising from high input power has not been previously quantified. To tackle this problem, we have performed one- and two-dimensional fully kinetic particle-in-cell simulations of the two possible coupling mechanisms, namely X-B and O-X-B mode conversion. We find that the ion dynamics has a profound effect on the field structure in the nonlinear regime, as high amplitude short-scale oscillations of the longitudinal electric field are excited in the region below the high-density cut-off prior to the arrival of the EBW. We identify this effect as the instability of the X wave with respect to resonant scattering into an EBW and a lower-hybrid wave. We calculate the instability rate analytically and find this basic theory to be in reasonable agreement with our simulation results.