Determinism and contingency shape evolutionary trajectories and the extent to which evolution is predictable depends on the relative contribution of each. However, the causes of predictable evolution are poorly understood. Here we take advantage of the fact that gene family recruitment may be more deterministic, and therefore predictable, when functional evolution is constrained to fewer genetic solutions. At the same time, the number of genes in a family capable of a specific function may be contingent on the vagaries of gene duplication mutations, functional changes, and maintenance, and therefore more difficult to predict. Here we analyze sulfotransferases using phylogenetics, gene expression, recombinant protein expression, and in vitro functional analyses. Sulfotransferases are the only enzymes known to catalyze sulfation, acting across a broad range of substrates and taxa, predicting that biological sulfation is catalyzed by sulfotransferases. We examine the bioluminescence system of ostracods and synthesize our findings with those reported previously for fireflies and sea pansies to show one aspect of bioluminescence, substrate metabolism, evolved convergently, each time recruiting sulfotransferase genes. However, our findings suggest that unlike fireflies, which have one known sulfotransferase gene specialized for substrate metabolism, ostracods contain several highly expressed sulfotransferase genes biochemically capable of metabolizing their bioluminescent substrate. While gene family recruitment may be predictable when limited genetic solutions exist, the number and specific identity of genes encoding a biological function may be difficult to predict because of the stochastic nature of gene duplication mutations and the diversity of mechanisms that fix and functionally differentiate these genes.