Pancreatic islet cells derived from human pluripotent stem cells hold great promise for modeling and treating diabetes. Differences between stem-cell-derived and primary islets remain, but molecular insights to inform improvements are limited. Here, we acquire single-cell transcriptomes and accessible chromatin profiles during in vitro islet differentiation and pancreas from childhood and adult donors for comparison. We delineate major cell types, define their regulomes, and describe spatiotemporal gene regulatory relationships between transcription factors. CDX2 emerged as a regulator of enterochromaffin-like cells, which we show resemble a transient, previously unrecognized, serotonin-producing pre-β cell population in fetal pancreas, arguing against a proposed non-pancreatic origin. Furthermore, we observe insufficient activation of signal-dependent transcriptional programs during in vitro β cell maturation and identify sex hormones as drivers of β cell proliferation in childhood. Altogether, our analysis provides a comprehensive understanding of cell fate acquisition in stem-cell-derived islets and a framework for manipulating cell identities and maturity.