We present RIPMMARC (Rotation Invariant Patch-based Multi-Modality Analysis aRChitecture), a flexible and widely applicable method for extracting information unique to a given modality from a multi-modal data set. We use RIPMMARC to improve the interpretation of arterial spin labeling (ASL) perfusion images by removing the component of perfusion that is predicted by the underlying anatomy. Using patch-based, rotation invariant descriptors derived from the anatomical image, we learn a predictive relationship between local neuroanatomical structure and the corresponding perfusion image. This relation allows us to produce an image of perfusion that would be predicted given only the underlying anatomy and a residual image that represents perfusion information that cannot be predicted by anatomical features. Our learned structural features are significantly better at predicting brain perfusion than tissue probability maps, which are the input to standard partial volume correction techniques. Studies in test-retest data show that both the anatomically predicted and residual perfusion signals are highly replicable for a given subject. In a pediatric population, both the raw perfusion and structurally predicted images are tightly linked to age throughout adolescence throughout the brain. Interestingly, the residual perfusion also shows a strong correlation with age in selected regions including the hippocampi (corr = 0.38, p-value <10(-6)), precuneus (corr = -0.44, p < 10(-5)), and combined default mode network regions (corr = -0.45, p < 10(-8)) that is independent of global anatomy-perfusion trends. This finding suggests that there is a regionally heterogeneous pattern of functional specialization that is distinct from that of cortical structural development.