Damage to the Medial Temporal Lobe (MTL) impairs declar- ative memory and perception. The Representational-Hierar- chical (RH) Account explains such impairments by assuming that MTL stores conjunctive representations of items and events, and that individuals with MTL damage must rely upon representations of simple visual features in posterior visual cortex. A recent study revealed a surprising anti-perceptual learning effect in MTL-damaged individuals: with exposure to a set of visual stimuli, discrimination performance worsened rather than improved. We expand the RH account to explain this paradox by assuming that visual discrimination is per- formed using a familiarity heuristic. Exposure to a set of highly similar stimuli entails repeated presentation of simple visual features, eventually rendering all feature representations equally (maximally) familiar and hence inutile for solving the task. Since the unique conjunctions represented in MTL do not occur repeatedly, healthy individuals are shielded from this perceptual interference. We simulate this mechanism with a neural network previously used to simulate recognition memory, thereby providing a model that accounts for both mnemonic and perceptual deficits caused by MTL damage us- ing a unified architecture and mechanism.