Apolipoprotein (apo) E4 is the major genetic risk factor for Alzheimer's disease (AD), increasing risk and decreasing age of disease onset. Many studies have demonstrated the detrimental effects of apoE4 in varying cellular contexts. However, the underlying mechanisms explaining how apoE4 leads to cognitive decline are not fully understood. Recently, the combination of human induced pluripotent stem cell (hiPSC) modeling of neurological diseases in vitro and electrophysiological studies in vivo have begun to unravel the intersection between apoE4, neuronal subtype dysfunction or loss, subsequent network deficits, and eventual cognitive decline. In this review, we provide an overview of the literature describing apoE4's detrimental effects in the central nervous system (CNS), specifically focusing on its contribution to neuronal subtype dysfunction or loss. We focus on γ-aminobutyric acid (GABA)-expressing interneurons in the hippocampus, which are selectively vulnerable to apoE4-mediated neurotoxicity. Additionally, we discuss the importance of the GABAergic inhibitory network to proper cognitive function and how dysfunction of this network manifests in AD. Finally, we examine how apoE4-mediated GABAergic interneuron loss can lead to inhibitory network deficits and how this deficit results in cognitive decline. We propose the following working model: Aging and/or stress induces neuronal expression of apoE. GABAergic interneurons are selectively vulnerable to intracellularly produced apoE4, through a tau dependent mechanism, which leads to their dysfunction and eventual death. In turn, GABAergic interneuron loss causes hyperexcitability and dysregulation of neural networks in the hippocampus and cortex. This dysfunction results in learning, memory, and other cognitive deficits that are the central features of AD.