- Main
Solution Path Clustering with Minimax Concave Penalty and Its Applications to Noisy Big Data
- Marchetti, Yuliya
- Advisor(s): Zhou, Qing
Abstract
Fast accumulation of large amounts of complex data has created a need
for more sophisticated statistical methodologies to discover
interesting patterns and better extract information from these data.
The large scale of the data often results in challenging
high-dimensional estimation problems where only a minority of the data
shows specific grouping patterns. To address these emerging challenges,
we develop a new clustering methodology that introduces the idea of a
regularization path into unsupervised learning. A regularization path
for a clustering problem is created by varying the degree of sparsity
constraint that is imposed on the differences between objects via the
minimax concave penalty with adaptive tuning parameters. Instead of
providing a single solution represented by a cluster assignment for
each object, the method produces a short sequence of solutions that
determines not only the cluster assignment but also a corresponding
number of clusters for each solution. The optimization of the penalized
loss function is carried out through an MM algorithm with block
coordinate descent. The advantages of this clustering algorithm
compared to other existing methods are as follows: it does not require
the input of the number of clusters; it is capable of simultaneously
separating irrelevant or noisy observations that show no grouping
pattern, which can greatly improve data interpretation; and it is a general
methodology that can be applied to many clustering problems. We then develop
an iterative subsampling approach to improve the computational efficiency
of this clustering methodology. The proposed approach iterates between
clustering a small subsample of the full data and sequentially
assigning the other data points to attain orders of magnitude of
computational savings. It preserves the
ability to isolate noise, includes a solution selection mechanism that
ultimately provides one clustering solution
with an estimated number of clusters, and is shown to be able to extract small
tight clusters from noisy data. The iterative subsampling approach's relatively minor losses in
accuracy are demonstrated through simulation studies, and its ability to handle large
datasets is illustrated through applications to gene expression datasets.
Main Content
Enter the password to open this PDF file:
-
-
-
-
-
-
-
-
-
-
-
-
-
-