The atomic scale structure of the active sites in heterogeneous catalysts is central to their reactivity and selectivity. Therefore, understanding active site stability and evolution under different reaction conditions is key to the design of efficient and robust catalysts. Herein we describe theoretical calculations which predict that carbon monoxide can be used to stabilize different active site geometries in bimetallic alloys and then demonstrate experimentally that the same PdAu bimetallic catalyst can be transitioned between a single-atom alloy and a Pd cluster phase. Each state of the catalyst exhibits distinct selectivity for the dehydrogenation of ethanol reaction with the single-atom alloy phase exhibiting high selectivity to acetaldehyde and hydrogen versus a range of products from Pd clusters. First-principles based Monte Carlo calculations explain the origin of this active site ensemble size tuning effect, and this work serves as a demonstration of what should be a general phenomenon that enables in situ control over catalyst selectivity.