Binocular rivalry is perceptual alternation that occurs when different visual images are presented to each eye. Despite the intensive studies, the mechanism of binocular rivalry still remains unclear. In multistable binocular rivalry, which is a special case of binocular rivalry, it is known that the perceptual alternation between paired patterns is more frequent than that between unpaired patterns. This result suggests that perceptual transition in binocular rivalry is not a simple random process, and the memories stored in the brain can play an important role in the perceptual transition. In this study, we propose a hierarchical chaotic neural network model for multistable binocular rivalry and show that our model reproduces some characteristic features observed in multistable binocular rivalry.