Aphantasia, namely the inability to voluntarily form visual mental imagery, does not, counterintuitively, impair the affected from successfully performing mental imagery tasks. One way of explaining this finding is to posit that aphantasics, despite their claim to the contrary, can form visual imagery, a position here referred to as aphantasia skepticism. This article outlines and rejects two types of aphantasia skepticism and argues that the position results from what is coined the visualizer's fallacy, namely the false belief that visual mental imagery is necessary to carry out mental imagery tasks. Furthermore, it is argued that the visualizer's fallacy and the resulting aphantasia skepticism are not only potentially harmful to aphantasics but may also lead to an impoverished view of the dynamics of cognition in general.