The void mediated formation mechanism of Sn quantum dots in an Si matrix was analyzed. It was found that the diffusion of Sn atoms into these voids leads to an initial rapid coarsening of the quantum dots during annealing. However, as this process is not restricted to Sn, the quantum dots could be grown by controlling the formation of voids and the diffusion of materials in these voids during molecular beam epitaxy.