© 2017, The Author(s). Relativistic quantum systems that admit scattering experiments are quantitatively described by effective field theories, where S-matrix kinematics and symmetry considerations are encoded in the operator spectrum of the EFT. In this paper we use the S-matrix to derive the structure of the EFT operator basis, providing complementary descriptions in (i) position space utilizing the conformal algebra and cohomology and (ii) momentum space via an algebraic formulation in terms of a ring of momenta with kinematics implemented as an ideal. These frameworks systematically handle redundancies associated with equations of motion (on-shell) and integration by parts (momentum conservation). We introduce a partition function, termed the Hilbert series, to enumerate the operator basis — correspondingly, the S-matrix — and derive a matrix integral expression to compute the Hilbert series. The expression is general, easily applied in any spacetime dimension, with arbitrary field content and (linearly realized) symmetries. In addition to counting, we discuss construction of the basis. Simple algorithms follow from the algebraic formulation in momentum space. We explicitly compute the basis for operators involving up to n = 5 scalar fields. This construction universally applies to fields with spin, since the operator basis for scalars encodes the momentum dependence of n-point amplitudes. We discuss in detail the operator basis for non-linearly realized symmetries. In the presence of massless particles, there is freedom to impose additional structure on the S- matrix in the form of soft limits. The most na¨ıve implementation for massless scalars leads to the operator basis for pions, which we confirm using the standard CCWZ formulation for non-linear realizations. Although primarily discussed in the language of EFT, some of our results — conceptual and quantitative — may be of broader use in studying conformal field theories as well as the AdS/CFT correspondence.