Finding a basis/coordinate system that can efficiently represent an input data
stream by viewing them as realizations of a stochastic process is of tremendous importance
in many fields including data compression and computational neuroscience. Two popular
measures of such efficiency of a basis are sparsity (measured by the expected $\ell^p$
norm, $0 < p \leq 1$) and statistical independence (measured by the mutual information).
Gaining deeper understanding of their intricate relationship, however, remains elusive.
Therefore, we chose to study a simple synthetic stochastic process called the spike
process, which puts a unit impulse at a random location in an $n$-dimensional vector for
each realization. For this process, we obtained the following results: 1) The standard
basis is the best both in terms of sparsity and statistical independence if $n \geq 5$ and
the search of basis is restricted within all possible orthonormal bases in $R^n$; 2) If we
extend our basis search in all possible invertible linear transformations in $R^n$, then
the best basis in statistical independence differs from the one in sparsity; 3) In either
of the above, the best basis in statistical independence is not unique, and there even
exist those which make the inputs completely dense; 4) There is no linear invertible
transformation that achieves the true statistical independence for $n > 2$.