The membrane structure has been applied throughout different fields such as civil engineering, biology, and aeronautics, among others. In many applications, large deflections negate linearizing assumptions, and linear modes begin to interact due to the nonlinearity. This paper considers the coupling effect between vibration modes and develops the theoretical analysis of the free vibration problem for orthotropic rectangular membrane structures. Von Kármán theory is applied to model the nonlinear dynamics of these membrane structures with sufficiently large deformation. The transverse displacement fields are expanded with both symmetric and asymmetric modes, and the stress function form is built with these coupled modes. Then, a reduced model with a set of coupled equations may be obtained by the Galerkin technique, which is then solved numerically by the fourth-order Runge-Kutta method. The model is validated by means of an experimental study. The proposed model can be used to quantitatively predict the softening behavior of amplitude-frequency, confirm the asymmetric characters of mode space distribution, and reveal the influence of various geometric and material parameters on the nonlinear dynamics.