Transcription factors from the NusG family bind to the elongating RNA polymerase to enable synthesis of long RNAs in all domains of life. In bacteria, NusG frequently co-exists with specialized paralogs that regulate expression of a small set of targets, many of which encode virulence factors. Escherichia coli RfaH is the exemplar of this regulatory mechanism. In contrast to NusG, which freely binds to RNA polymerase, RfaH exists in a structurally distinct autoinhibitory state in which the RNA polymerase-binding site is buried at the interface between two RfaH domains. Binding to an ops DNA sequence triggers structural transformation wherein the domains dissociate and RfaH refolds into a NusG-like structure. Formation of the autoinhibitory state, and thus sequence-specific recruitment, represents the decisive step in the evolutionary history of the RfaH subfamily. We used computational and experimental approaches to identify the residues that confer the unique regulatory properties of RfaH. Our analysis highlighted highly conserved Ile and Phe residues at the RfaH interdomain interface. Replacement of these residues with equally conserved Glu and Val counterpart residues in NusG destabilized interactions between the RfaH domains and allowed sequence-independent recruitment to RNA polymerase, suggesting a plausible pathway for diversification of NusG paralogs.