Grapheme-color synesthetes experience linguistic symbols as having a consistent color (e.g., “The letter R is burgundy.”). Intriguingly, certain letters tend to be associated with certain colors, and these biases are not random: numerous properties of letters influence which letter is associated with which color. These influences, called “Regulatory Factors” (RFs), each explain some fraction of the variation in observed associations. No comprehensive model of the influences on grapheme-color associations exists: RFs have only been measured in isolation, are not always operationalized consistently, and often make competing predictions that cannot be accounted for in a univariate model. Here, we describe a statistical framework that integrates the predictions of all RFs into a single model, and thus yields a unified account of their influence on grapheme-color associations. Our model also links these predictions to measurable properties of language, offering a window into the multifactorial contributions to letter representation in the brain.