Computer vision can be considered a highly specialized data collection and data analysis problem. We need to understand the special properties of image data in order to construct statistical models for representing the wide variety of image patterns. One special property of vision that distinguishes itself from other sensory data such as speech data is that distance or scale plays a profound role in image data. More specifically, visual objects and patterns can appear at a wide range of distances or scales, and the same visual pattern appearing at different distances or scales produces different image data with different statistical properties, thus entails different regimes of statistical models. In particular, we show that the entropy rate of the image data changes over the viewing distance (as well as the camera resolution). Moreover, the inferential uncertainty changes with viewing distance too. We call these changes information scaling. From this perspective, we examine both empirically and theoretically two prominent and yet largely isolated research themes in image modeling literature, namely, wavelet sparse coding and Markov random fields. Our results indicate that the two models are appropriate on two different entropy regimes: sparse coding targets the low entropy regime, whereas the random fields are suitable for the high entropy regime. Because of information scaling, both models are necessary for representing and interpreting image intensity patterns in the whole entropy range, and information scaling triggers transitions between these two regimes of models. This motivates us to propose a full-zoom primal sketch model that integrates both sparse coding and Markov random fields. In this model, local image intensity patterns are classified into “sketchable regime” and “non-sketchable regime” by a sketchability criterion. In the sketchable regime, the image data are represented deterministically by highly parametrized sketch primitives. In the non-sketchable regime, the image data are characterized by Markov random fields whose sufficient statistics summarize computational results from failed attempts of sparse coding. The contribution of our work is two folded. First, information scaling provides a dimension to chart the space of natural images. Second, the full-zoom modeling scheme provides a natural integration of sparse coding and Markov random fields, thus enables us to develop a new and richer class of statistical models.

This paper presents a mathematical framework for visual learning that integrates two popular statistical learning paradigms in the literature: I) Descriptive learning, such as Markov random fields and minimax entropy learning, and II) Generative learning, such as PCA, ICA, TCA, and HMM. We apply the integrated learning framework to texture modeling, and we assume that an observed texture image is generated by multiple layers of hidden stochastic processes with various texture elements called "textons". Each texton is expressed as a window function like a mini-template or a wavelet, and each hidden stochastic process is a spatial pattern with a number of textons subject to affine transfomrations. The hidden layers are characterized by minimax entropy models, and they generate images by occlusion or linear addition. Thus given a raw input image, the learning framework achieves four goals: i) Computing the appearance of textons. ii) Inferring the hidden stochastic processes iii) Learning Gibbs models for each hidden stochastic process and iv) Verifying the learnt textons and models through random sampling. The integrated framework subsumes the minimax entropy learning paradigm and creates a richer class of probability models for visual patterns. Furthermore we show that the integration of descriptive and generative methods is a natural path of visual learning. We demonstrate the proposed framework and algorithms on many real images.

Recent results on sparse coding and independent component analysis suggest that human vision first represents a visual image by a linear superposition of a relatively small number of localized elongate, oriented image bases. With this representation, the sketch of an image consists of the locations, orientations, and elongations of the images bases, and the sketch can be visually illustrated by depicting each image base by a linelet of the same length and orientation. Built on the insight of sparse and independent component analysis, we propose a two-level generative model for textures. At the bottom-level, the texture image is represented by a linear superposition of image bases. At the top-level, a Markov model is assumed for the placement of the image bases or the sketch, and the model is characterized by a set of simple geometrical feature statistics.

Vision can be posed as a statistical learning and inference problem. As an over-simplified account, let W be a description of the outside scene in terms of ìwhat is where,î let I be the retina image, and let p(W, I) be the joint distribution of W and I. 1 Then visual learning is to learn p(W, I) from training data, and visual perception is to infer W from I based on p(W|I).

There are two major schools on visual learning and perception. One school is operation-oriented and learns the inferential process defined by p(W|I) directly, often in the form of an explicit transformation W ? F(I). This scheme is mostly used in supervised learning, where W is object category, and is given in training data. The other school is representation-oriented and learns the generative process p(W) and p(I|W) explicitly, then perception is to invert the generative process by maximizing or sampling p(W|I) ? p(W)p(I|W). In this scheme, p(W) may also be accounted for by a regularization term such as smoothness or sparsity. This scheme is often used in unsupervised learning where W is not available in training data.

Following Marr’s insight, we propose a generative image representation called primal sketch, which integrates two modeling components. The first component explains the structural part of an image, such as ob ject boundaries, by a hidden layer of image primitives. The second component models the remaining textural part without distinguishable elements by Markov random fields that interpolate the structural part of the image. We adopt an artist’s notion by calling the two components “sketchable” and “non-sketchable” parts respectively. A dictionary of image primitives are used for modeling structures in natural images, and each primitive is specified by variables for its photometric, geometric, and topological attributes. The primitives in the image representation are not independent but organized in an sketch graph. This sketch graph is modeled by a spatial Markov model that enforces Gestalt organizations. The inference of the sketch graph consists of two phases. Phase I sequentially adds the most prominent image primitives in a procedure similar to matching pursuit. Phase II edits the sketch graph by a number of graph operators to achieve good Gestalt organizations. Experiments show that the primal sketch model produces satisfactory results for a large number of generic images. The primal sketch model is not only a parsimonious image representation for lossy image coding, but also provides a meaningful mid-level generic representation for other vision tasks.

Textons refer to fundamental micro-structures in generic natural images and thus constitute the basic elements in early (pre-attentive) visual perception. However, the word "textong" remains a vague concept in the literature of computer vision and visual perception, and a precist mathematical definition has yet to be found. In this article, we argue that the definition of texton should be governed by a sound mathematical model of images, and the set of textons must be learned from, or best tuned to, an image ensemble. We adopt a generative image model that an image is a superposition of bases from an over-complete dictionary, then a texton is defined as a mini-template that consists of a varying number of image bases with some geometric and photometric configurations. By analogy to physics, if image bases are like protons, neutrons and electrons, then textons are like atoms. Then a small number of textons can be learned from training images as repeating micro-structures. We report four experiments for comparison. The first experiment computes clusters in feature space of filter responses. The second transformed component analysis in both feature space and image patches. The third adopts a two-layer generative model where an image is generated by image bases and image bases are generated by textons. The fourth experiment shows textons from motion image sequences, which we call movetons.