Heterosis denotes the superiority of a hybrid plant over its parents. The use of heterosis has contributed significantly to yield improvement in crops. However, the genetic and molecular bases on heterosis are not fully understood. A large number of heterotic loci were identified for 12 yield-related traits in one parental population of chromosome segment substitution lines (CSSLs) and two test populations, which were interconnected by CSSLs derived from two rice genome-sequenced cultivars, Nipponbare and Zhenshan 97. Seventy-five heterotic loci were identified in both homozygous background of Zhenshan 97 and heterogeneous background of an elite hybrid cultivar Shanyou 63. Among the detected loci, at least 11 were colocalized in the same regions encompassing previously reported heterosis-associated genes. Furthermore, a heterotic locus Ghd8NIP for yield advantage was verified using transgenic experiments. Various allelic interaction at Ghd8 exhibited different heterosis levels in hetero-allelic combinations of five near-isogenic lines that contain a particular allele. The significant overdominance effects from some hetero-allelic combinations were found to improve yield heterosis in hybrid cultivars. Our findings support the role of allelic interaction at heterotic loci in the improvement of yield potential, which will be helpful for dissecting the genetic basis of heterosis and provide an optional strategy for the allele replacement in molecular breeding programs in hybrid rice.