UCLA

Fast accumulation of large amounts of complex data has created a need

for more sophisticated statistical methodologies to discover

interesting patterns and better extract information from these data.

The large scale of the data often results in challenging

high-dimensional estimation problems where only a minority of the data

shows specific grouping patterns. To address these emerging challenges,

we develop a new clustering methodology that introduces the idea of a

regularization path into unsupervised learning. A regularization path

for a clustering problem is created by varying the degree of sparsity

constraint that is imposed on the differences between objects via the

minimax concave penalty with adaptive tuning parameters. Instead of

providing a single solution represented by a cluster assignment for

each object, the method produces a short sequence of solutions that

determines not only the cluster assignment but also a corresponding

number of clusters for each solution. The optimization of the penalized

loss function is carried out through an MM algorithm with block

coordinate descent. The advantages of this clustering algorithm

compared to other existing methods are as follows: it does not require

the input of the number of clusters; it is capable of simultaneously

separating irrelevant or noisy observations that show no grouping

pattern, which can greatly improve data interpretation; and it is a general

methodology that can be applied to many clustering problems. We then develop

an iterative subsampling approach to improve the computational efficiency

of this clustering methodology. The proposed approach iterates between

clustering a small subsample of the full data and sequentially

assigning the other data points to attain orders of magnitude of

computational savings. It preserves the

ability to isolate noise, includes a solution selection mechanism that

ultimately provides one clustering solution

with an estimated number of clusters, and is shown to be able to extract small

tight clusters from noisy data. The iterative subsampling approach's relatively minor losses in

accuracy are demonstrated through simulation studies, and its ability to handle large

datasets is illustrated through applications to gene expression datasets.