Skip to main content
eScholarship
Open Access Publications from the University of California

The Origin of Clusters in Recurrent Neural Network State Space

Abstract

Cluster analysis has been successfully applied to the problem understanding hidden unit representations in both feed-forward and recurrent neural networks. While the topological properties of feed-forward networks may support the use of cluster analysis, the results described within this paper suggest that applications to recurrent networks are not justified. This paper illustrates how clustering fails to provide useful insights into the underlying task-dependent information processing mechanism of recurrent networks. In this paper, I first demonstrate that randomly generated networks display a surprising amount of clustering before training. Then I explain that the clustering structure emerges, not in response to the task training, but because of the volume-reducing iterated mappings that comprise the commonly used recurrent neural networks models.

Main Content
For improved accessibility of PDF content, download the file to your device.
Current View