The voltage-gated ion channel activity depends on both activation (transition from the resting state to the open state) and inactivation. Inactivation is a self-restraint mechanism to limit ion conduction and is as crucial to membrane excitability as activation. Inactivation can occur when the channel is open or closed. Although open-state inactivation is well understood, the molecular basis of closed-state inactivation has remained elusive. We report cryo-EM structures of human KV4.2 channel complexes in inactivated, open, and closed states. Closed-state inactivation of KV4 involves an unprecedented symmetry breakdown for pore closure by only two of the four S4-S5 linkers, distinct from known mechanisms of open-state inactivation. We further capture KV4 in a putative resting state, revealing how voltage sensor movements control the pore. Moreover, our structures provide insights regarding channel modulation by KChIP2 and DPP6 auxiliary subunits. Our findings elucidate mechanisms of closed-state inactivation and voltage-dependent activation of the KV4 channel.