We study behavior of the restricted maximum likelihood (REML) estimator under a
misspecified linear mixed model (LMM) that has received much attention in recent gnome-wide
association studies. The asymptotic analysis establishes consistency of the REML estimator
of the variance of the errors in the LMM, and convergence in probability of the REML
estimator of the variance of the random effects in the LMM to a certain limit, which is
equal to the true variance of the random effects multiplied by the limiting proportion of
the nonzero random effects present in the LMM. The aymptotic results also establish
convergence rate (in probability) of the REML estimators as well as a result regarding
convergence of the asymptotic conditional variance of the REML estimator. The asymptotic
results are fully supported by the results of empirical studies, which include extensive
simulation studies that compare the performance of the REML estimator (under the
misspecified LMM) with other existing methods.