We investigated individuals’ representation of composite numbers (i.e., those with factors other than 1 and themselves). Some numbers are “more composite” (have more factors) than others (e.g., 72 vs. 74). We hypothesized that people mentally represent numbers in factor neighborhoods. This should result in a compositeness penalty: performance on factoring problems (e.g., x * 9 = 72) should be slower for more composite products, because retrieving the corresponding factor requires inhibiting neighboring factor pairs. We measured 50 undergraduates’ time to correctly solve factoring problems for composite products (6 to 100). We found that for well-practiced facts (i.e., the 12x12 multiplication table), performance slowed as compositeness increased, even after controlling for problem size and factor size. However, there was no compositeness penalty for “non-table” facts. These findings support our factor neighborhood hypothesis for well-practiced facts and suggest that people's representations of composite numbers are influenced by mathematics instruction (i.e., “multiplication tables”).