An object-based approach for verification of precipitation estimation

Verification has become an integral component in the development of precipitation algorithms used in satellite-based precipitation products and evaluation of numerical weather prediction models. A number of object-based verification methods have been developed to quantify the errors related to spatial patterns and placement of precipitation. In this study, an image processing technique known as watershed transformation, capable of detecting closely spaced, but separable precipitation areas, is adopted in the object-based approach. Several key attributes of the segmented precipitation objects are selected and interest values of those attributes are estimated based on the distance measurement of the estimated and reference images. An overall interest score is estimated from all the selected attributes and their interest values. The proposed object-based approach is implemented to validate satellite-based precipitation estimation against ground radar observations. The results indicate that the watershed segmentation technique is capable of separating the closely spaced local-scale precipitation areas. In addition, three verification metrics, including the object-based false alarm ratio, object-based missing ratio, and overall interest score, reveal the skill of precipitation estimates in depicting the spatial and geometric characteristics of the precipitation structure against observations.


Introduction
Accurate representation of observed precipitation spatial patterns and structures is essential for hydrologic applications. It has been noted that the spatial variability of precipitation has a major impact on the accuracy of modelled runoff volumes (Faurès et al. 1995;Goodrich et al. 1995). Especially in distributed hydrological modelling, the spatial patterns and locations of precipitation events are important to describe the spatial heterogeneity of precipitation (Foufoula-Georgiou and Vuruputur 2001).
In the last decade, satellite-based precipitation products and numerical weather prediction (NWP) models have provided precipitation estimates and forecasts, respectively, with high spatial and temporal resolution suitable for hydrologic modelling and watershed management. However, the quality of the simulated precipitation datasets is a vital factor in the decision to use these estimates for practical applications. Therefore, it is imperative that verification be an integral component of precipitation algorithms and dataset development.
Several coordinated verification activities have been established to evaluate the accuracy of precipitation estimation against ground observations, such as ground radar and rain gauge data (Adler et al. 2001;AghaKouchak et al. 2012;AghaKouchak, Behrangi, et al. 2011;Arkin and Turk 2006;Arkin and Xie 1994;Colle, Olson, and Tongue 2003;Ebert, Janowiak, and Kidd 2007;Ebert et al. 2003;Mass et al. 2002;McBride and Ebert 2000;Mehran and AghaKouchak 2014;Olson, Junker, and Korty 1995;Sapiano and Arkin 2009;Smith et al. 1998;Tian et al. 2009). In general, these activities have focused on pixel-based measures of errors, including continuous metrics (e.g. correlation coefficient, root mean square error) and/or categorical statistics (e.g. probability of detection (POD), false alarm ratio (FAR), critical success index (CSI) derived from contingency tables; Wilks (2011)). Volumetric versions of the above metrics have also been developed and used in validation and verification studies (AghaKouchak and Mehran 2013). These verification measures summarize pixel-to-pixel differences between observations and estimates. However, while providing valuable information, these measures do not explicitly quantify errors related to spatial patterns, structure, and placement of precipitation (Baldwin and Kain 2006;Casati et al. 2008), which are fundamental in hydrologic applications. Figure 1 illustrates the inability of traditional verification methods to capture spatial characteristics of precipitation errors. Five examples of observation (O) and estimate/forecast (E) pairs are presented in Figure 1. The first four examples yield the same statistics: POD = 0, FAR = 1, CSI = 0, indicating that the estimate/forecast completely fails to capture the event. However, visual inspection of Figure 1 shows that in case (a), the estimate (E) captures the observation (O), but with a slight displacement. Case (b) shows larger displacement, but (E) still manages to capture the shape of the storm. Cases (c) and (d) poorly capture the observation, with (c) overestimating the area and (d) misorientating the estimate (E). Case (e) scores better in traditional verification metrics (POD > 0, FAR < 1, CSI > 0), but is probably a poorer estimate than (a). Thus, these verification statistics are not able to diagnose spatial and geometrical features of errors.
A number of spatial verification methods have been proposed to provide diagnostic information regarding the ability of a NWP model to depict the spatial pattern, intensity, and placement of precipitation entities compared with the observations (see Ahijevych et al. (2009, and references therein). Furthermore, Gilleland et al. (2009Gilleland et al. ( , 2010 categorized the spatial verification methods into four groups: (1) neighbourhood approaches, (2) scale separation/decomposition, (3) features-based/objectbased approaches, and (4) field deformation verification. These groups have been further compared and examined for high-resolution precipitation forecasts. For example, in the object-based approaches, Ebert and McBride (2000) and Ebert and Gallus (2009) developed the contiguous rain area (CRA) method, in which the total mean squared error is decomposed into volume, pattern, and displacement error. Davis et al. (2006aDavis et al. ( , 2006bDavis et al. ( , 2009 introduced the method for object-based diagnostic evaluation (MODE), which focuses on multiple attributes of rain entities (e.g. orientation, rain area) to derive the median of maximum interest (i.e. an object-based measure of forecast skill) using a fuzzy logic algorithm. Wernli et al. (2008Wernli et al. ( , 2009) considered three components: structure (S), amplitude (A), and location (L), associated with precipitation fields in the verification domain. Sandgathe (2006, 2008) Figure 1. A schematic instance of various observation (O) and estimate/forecast (E) combinations (Davis, Brown, and Bullock 2006a). to identify objects and assessed the forecast performance on different scales. Micheas et al. (2007) proposed Procrustes shape analysis to evaluate the forecast skill. Lack, Limpert, and Fox (2010) advanced Micheas et al. (2007)'s method by using a discrete Fourier transform to allow object identification on multiple scales.

adopted a statistical method termed cluster analysis
In the advanced concept workshop on remote sensing of precipitation on multiple scales , further study of diagnostic techniques such as objectbased verification approaches was identified as one of the research priorities in the remote sensing of precipitation. A number of object-based verification and pattern analysis approaches have been implemented in satellite precipitation estimates (e.g. AghaKouchak, Nasrollahi, et al. 2011;Demaria et al. 2011). For instance, Skok et al. (2009) adopted MODE to compare the spatial distribution and movement of precipitation systems derived from tropical rainfall measuring mission (TRMM) 3B42 and precipitation estimation from remotely sensed information using artificial neural networks (PERSIANN) over the intertropical convergence zone. In addition, Demaria et al. (2011) examined the systematic errors related to volume, pattern, and displacement using the CRA method for three satellite precipitation products: TRMM, PERSIANN, and the climate prediction centre morphing technique (CMORPH), against rain gauge observations in the La Plata river basin.
Most object-based methods use a threshold to define objects as contiguous regions of pixels that exceed the specified threshold ). A schematic onedimensional example of using a threshold in the precipitation field is shown in Figure 2. Two peaks of precipitation intensity are observed in the example, which can indicate two localized precipitation areas in the field. Using the high threshold 'a', two precipitation objects can be identified, but only a few points are above the threshold and are included in each object. Using the low threshold 'c', one precipitation object can be identified. This large object is useful when analysing large-scale or meso-scale precipitation systems. Using the threshold 'b', two objects with many points included can be identified. This threshold is an appropriate threshold to distinguish localized precipitation areas. However, the appropriate threshold needs to be selected from a specific range in order to separate closely spaced precipitation areas. This increases the difficulty in identifying localized precipitation areas using the thresholding technique. Meanwhile, localized precipitation areas are found to possess different spatial characteristics, such as sizes, shapes, and orientations. Also, they are found to experience differences in the temporal domain, such as different advection, and growth/decay rates. In this article, an advanced object identification technique is proposed for the spatial verification of  Figure 2. A schematic example of using different thresholds in the precipitation field in one dimension. The precipitation intensity function is defined on the x-y plane. Here, the example is based on one dimension x. The black curve represents the precipitation intensity. Grey dash lines represent different thresholds. localized precipitation areas. An image processing technique known as the watershed transformation (Beucher and Lantuejoul 1979;Meyer 1994) is adopted in the object-based approach to detect the closely spaced, but separable local-scale precipitation areas both in the estimations and observations. Lakshmanan, Hondl, and Rabin (2009) used the watershed transformation technique to identify storm cells in radar reflectivity and infrared temperature satellite data. This study employed the watershed transformation technique to identify the localized precipitation areas in the precipitation estimations and observations for the verification purpose, which is one of the main goals of this study.
The objective of this study is to develop an advanced technique for the verification of local-scale precipitation areas focusing on their spatial and geometric characteristics. There are already numerous categorical and volumetric verification measures in the literature (e.g. Wilks 2011; AghaKouchak and Mehran 2013). Thus, the focus of this study is on geometrical characterization. The proposed verification method uses the watershed transformation to identify and delineate the separable local-scale precipitation objects, and then uses a distance measurement for selected precipitation object attributes by estimating the interest values. The selected precipitation objects are used for relative comparison of the precipitation geometric characteristics, but not precipitation intensity.

Precipitation object identification
The first step is to identify separable local-scale precipitation objects. The watershed segmentation/transformation is the main algorithm used for identification of precipitation objects, which was developed by Meyer (1994). This algorithm partitions the image into different catchment basins/segmented regions based on local minima. The image is interpreted as a topographic surface, where the value of each pixel represents the altitude at that point. In this study, watershed transformation is adopted to segment precipitation intensity images based on storm centres. Since storm centres are associated with the highest rainfall intensity (i.e. they are local maxima), the precipitation data that possess storm centres are converted to possess local minima in order to apply the watershed transformation algorithm. This conversion of the precipitation data is described as: where p is the original precipitation intensity at each pixel and λ is a positive number that assures that p′ is a positive value for any given p.
Watershed transformation often leads to over-segmentation in the images due to numerous local minima (Meyer and Beucher 1990). Additionally, the local minima, which are minor fluctuations, are insignificant for the purpose of segmentation (Meyer 1994). The H-minima transform is used to suppress/fill the local shallow minima prior to segmentation (Soille 1999), where the local shallow minima are the local minima whose depths are smaller or equal to the merge threshold. These minima are filled by increasing the minima with the merge threshold and are suppressed in the images. Figure 3 shows the effect of suppressing the local shallow minima using the H-minima transform.
After using the H-minima transform, the watershed transformation algorithm is then applied to identify catchment basins. The algorithm is based on topographical distance. It is assumed that the image h is in the space C(E). This space represents the real twice continuously differentiable functions on the connected domain E with only isolated critical points (Roerdink and Meijster 2000). The topographical distance (TD) between two points, a and b in domain E, is defined as: where inf refers to infimum and γ represents all paths between a and b in domain E, with γ(0) = a, γ(1) = b. The shortest TD between a and b is obtained from the path that has the steepest slope. Given that {m k } k∈I is the set of local minima in h and I contains a set of indices k, the catchment basin, CB(m i ) with the minimum m i , is defined as the set of points x ∈ E, which are closer to m i than any other local minimum m j based on TD: The watershed line (WL) of h is the set of points that do not belong to any catchment basins: Every point in the WL is assigned to a certain CB based on the minimum value difference between this point and its eight neighbourhood points. Then, the watershed transformation algorithm assigns different labels to each CB. Catchment basins/storm objects are identified in the converted precipitation intensity images after applying the watershed transformation. Then, the minimum size threshold is used to decide the lower bound coverage of precipitation objects; the objects above the threshold will be further considered in the evaluation. The precipitation objects are compared for the precipitation estimates and the observations at the same merge and minimum size thresholds. The same thresholds are used to test the similarity between the estimated and observed precipitation fields. An example of the segmentation process of the precipitation objects is illustrated in Figure 4. This figure serves as an instance to    demonstrate how the merge threshold affects the resulting precipitation objects in the reference data (i.e. observations). Here, different merge thresholds were used to show the separation of precipitation areas. When the merge threshold is small (1 to 10 mm day −1 in this case), segmented precipitation objects tend to delineate small-scale precipitation objects. On the other hand, when the merge threshold is large (greater than 25 mm day −1 in this case), segmented precipitation objects tend to combine the closely spaced precipitation areas. Therefore, the appropriate merge threshold should be selected based on the scale at which validation information is required. Note that the merge threshold can be determined based on the user's particular application. If a user is interested in a more detailed structure/pattern of the precipitation field, a relatively lower merge threshold can be considered. If a user is interested in the large-scale structure/ pattern of the precipitation field, a relatively higher merge threshold can be implemented. In this study, a 15 mm day −1 threshold is used for a local-scale evaluation of satellite precipitation information.

Precipitation object evaluation
After identifying the precipitation objects, the next step is to evaluate how well the estimated objects match with the observed objects. The estimated and observed precipitation objects, obtained from estimation and observation fields, are categorized into matching objects and unmatching objects based on overlapping criteria. Each observed object is searched for an estimated matching object by examining the overlap between the two objects. The two objects can be partially overlapped or fully overlapped. Both situations are considered as matching cases. For the unmatching objects, two scores are generated to examine the ability of the estimation for detecting the precipitation events. The estimated objects that do not overlap with any observed objects are counted as false alarms; the object-based false alarm ratio (OFAR) is determined by dividing the false alarm object area by the total object area in the estimate. The same procedures are conducted on the observed objects. The observed objects that do not overlap with any estimated objects are counted as misses; and the object-based missing ratio (OMR) is determined by dividing the miss object area by the total object area in the observation. The two scores are defined as: OFAR ¼ False alarm objects area Total area of estimated objects ; OMR ¼ Miss objects area Total area of observed objects : It should be noted that the object-based false alarms and misses are different from the traditional pixel-based false alarms and misses, respectively. The object-based false alarms/misses include only the pixels that have not been matched as part of objects. Thus, the false alarm/miss object area tends to include fewer pixels than the traditional false alarm/miss pixels. OFAR and OMR relax the requirement of traditional pixelbased hits by allowing all the pixels that are within the matching objects to be considered as hits. For the matching objects, four key attributes are selected to examine the performance of the estimated precipitation objects for depicting the geometric characteristics against the observed precipitation objects. The attributes are the centroid location distance (cd), object area ratio (ar), intersection area ratio (iar), and orientation difference (od). They are defined as: where (x i , y i ) and (x j , y j ) are the coordinates of the geometrical centroids of the ith observed object and the jth estimated object, respectively. A i and A j are the area coverages of the ith observed object and the jth estimated object, respectively. A i;j is the overlap area between the pair of matching objects. δ i and δ j are the angles between the x-axis/ horizontal direction and the major axis of the fitted eclipse for the ith observed object and the jth estimated object, respectively. In addition, four interest functions (see Figure 5) are designed for each object attribute to obtain commensurable measures, termed interest values. The interest values, ranging from 0 to 1, can be aggregated and compared across object attributes. They represent how well the estimated object matched the observed object, and the highest interest value means a perfect performance of the estimated object for that specific attribute. Based on an attribute's histogram shown in Figure 6, a probability density function is created for each attribute. The extreme of 2.5% of each tail in the probability density function is given the worst or best interest values based on the attribute. Taking centroid distance for example, two extremes were discovered at 0.8 and 36 pixels. Note that one pixel unit is approximately 25 km in length. The interest value of 1 is assigned to the object pair whose centroid distance is smaller than 0.8 pixels, while the interest value of 0 is assigned to the object pair whose centroid distance is larger than 36 pixels. Here, the four interest functions are defined as: An overall interest score is generated for matching estimated/observed objects: where M i,j,cd , M i,j,ar , M i,j,iar , and M i,j,od are the interest values of attributes cd, ar, iar, and od between the ith observed and the jth estimated object pair. The terms w cd , w ar , w iar , and w od are the weights of each attribute, where the same weight of 25% is assigned in this study. o i;j is the overall interest value of the object pair. For each observed object, a weighted interest value, o i , is obtained based on the set of matching estimated objects (S i ). If the observed object matches with one estimated object, S i contains one matching estimated object and consequently o i is equal to o i;j . If the observed object matches  with multiple estimated objects, S i contains multiple matching estimated objects. o i is then calculated using the weights a i;j , with better matches getting a higher weight. The overall interest score, o, is the median of the weighted interest values o i from all of the matching observed objects. For a domain, the overall interest score, o, provides the information on the quality of the estimated data in an aggregated sense. In general, with a small centroid distance, a large intersection area ratio, a small object size difference, and a small orientation difference for the matching objects, the overall interest score would be high. On the other hand, with a large centroid distance, a small intersection area ratio, a large object size difference, and a large orientation difference, the overall interest score would be low.

Case study
In this section, the object-based approach described above is applied for validation of a satellite-based precipitation product against ground radar observations. It should be noted that the case study serves as an application of the framework to examine its feasibility.

Data
The satellite-based precipitation product used in this study is PERSIANN (Hsu et al. 1997;Sorooshian et al. 2000). It is a global precipitation estimation system using an artificial neural network approach. It estimates the rainfall rate at each 0.25°× 0.25°p ixel based on geostationary long-wave infrared imagery and is corrected by passive microwave information. NOAA stage IV MPE (Lin and Mitchell 2005) is used as the ground-observation reference. The stage IV data combines the precipitation measurements from radar and rain gauge at a 4 km spatial resolution. Here, the stage IV data are processed to derive the same spatial resolution as for the PERSIANN data. Four-kilometre grid stage IV data contained in a 0.25°× 0.25°pixel are averaged to obtain the stage IV data at a 0.25°spatial resolution. The case study is conducted at 0.25°× 0.25°on a daily scale in the summer of 2008 over the contiguous United States (CONUS). Here, daily precipitation estimates are selected for evaluation, because a large number of climate and hydrologic applications require daily precipitation estimates as their inputs. In addition, the reference data obtained from ground radar and rain gauges provide good quality at the 24-hour time scale. Therefore, the evaluation of precipitation estimates at the daily scale is performed in order to provide useful information about the skill of precipitation estimates for the hydrology community.  Figure 7(c) and (d) shows the precipitation objects identified using the convolution threshold technique (Davis et al. 2009). The convolution threshold technique uses the process of convolution and a threshold to identify objects in the precipitation estimation and observation. The process of convolution replaces the centred pixel value with the average over the window covered area. After convolution, the contiguous regions of pixels that exceed the threshold are defined as the precipitation objects. As seen in Figure 7(c) and (d), two dominant large rainfall objects over the middle and eastern USA are observed for stage IV and PERSIANN by using this thresholding approach. These rainfall objects represent synoptic scale precipitation areas and depict the precipitation structures in the synoptic scale. Figure 7(e) and (f) indicates how rainfall objects are shaped and distributed by implementing the watershed transformation on stage IV and PERSIANN with a merge threshold of 15 mm day −1 . Here, the segmented rainfall objects well represent the closely spaced, but separable local-scale 24-hour accumulated precipitation areas for stage IV and PERSIANN. The resulting rainfall objects are the separable local-scale precipitation areas that a human expert would likely see from the precipitation fields. This demonstrates the capability of the watershed transformation for separating the closely spaced mesoscale precipitation areas. The different precipitation objects delineate different precipitation areas generated from convective storms, and the precipitation object distribution depicts the accumulated precipitation structure in the local scale. In addition, PERSIANN precipitation objects are larger in size compared to stage IV precipitation objects. This is because PERSIANN tends to estimate larger precipitating areas as shown in Figure 7(b) compared to stage IV precipitating areas in Figure 7(a). PERSIANN uses the cloud-top brightness temperature to estimate the precipitation that results in producing a larger coverage for precipitation systems. This implies that the watershed transformation is able to preserve the original features of the precipitation in the segmented precipitation objects, which allows a diagnostic evaluation at the local scale.

Precipitation object evaluation
In the verification approach, the precipitation objects obtained from PERSIANN are evaluated against the precipitation objects obtained from stage IV. Figure 6 illustrates the histograms of four attributes in the summer of 2008, including centroid distance, area ratio, intersection area ratio, and orientation difference. In the centroid distance histogram, the number of matching object pairs decreases as the centroid distance increases. Most of the centroid distance is less than 20 pixels (one pixel unit is approximately 25 km in length), which indicates that the majority of matching objects have acceptable centroid distances. From the area ratio histogram, the peak is discovered at the value of 1. There are more object   pairs with area ratios (greater than 1) than the others. This shows that PERSIANN tends to generate larger size rain objects than stage IV. In the intersection area ratio histogram, the matching objects spread almost evenly between the lower bound and the higher bound. A peak is noted at the value of 1, representing perfect overlapping. Finally, from the orientation difference histogram, it is observed that the number of object pairs decreases as the orientation difference increases. Fifty-five per cent of the orientation difference is less than 30°, which indicates that PERSIANN performs well with regard to the orientation angle. Three verification scores, including the object-based false alarm ratio, missing ratio, and overall interest score, are derived using the proposed object-based approach. Figure 8 illustrates the time series of these scores for the summer of 2008. As seen in the figure, the object-based false alarm ratio and missing ratio are mostly below a value of 0.1. This indicates that PERSIANN is capable of detecting the local-scale precipitation areas correctly and placing them at the right location when compared with the stage IV observations. The overall interest score, summarized from four attributes of matching precipitation objects, shows good results throughout the entire summer. This indicates that PERSIANN has the ability to depict the similar characteristics of local-scale precipitation areas against reference data with respect to matching objects. Overall, the three scores reveal that PERSIANN has the skill to depict the spatial and geometric characteristics of the precipitation structure in the local scale compared to observations.

Summary and conclusions
An object-based approach is presented in this article to evaluate simulated precipitation by focusing on separable local-scale precipitation areas. The application of the approach is demonstrated in the validation of the satellite precipitation product PERSIANN against the stage IV rain analysis on a daily scale for the summer of 2008. The watershed transformation is adopted in the approach in order to identify the closely spaced, but separable local-scale precipitation areas. The application of watershed transformation allows the separation of closely spaced precipitation areas and identification of local-scale precipitation objects in the meantime. This assures that the precipitation object characteristics, acquired from separable localscale precipitation areas, are diagnostic and valuable for future evaluation. In addition, the three metrics, including the object-based false alarm ratio, missing ratio, and overall interest score, are examined to be informative and meaningful. These three scores are capable of demonstrating whether the simulated precipitation correctly detects the rainfall and how well the simulated precipitation can depict the spatial and geometric features of the precipitation structure. Hence, this object-based approach provides new insights into the object-based evaluation of precipitation estimation by characterizing the local-scale precipitation characteristics.
The proposed object-based approach is suggested for further implementation on other satellite-based precipitation products. The performance for depicting the spatial and geometric characteristics of the precipitation structure at the local-scale can be examined for different products on a daily scale. These evaluation activities using the object-based approach can be included in the satellite precipitation intercomparison project for a comprehensive understanding of different satellite-based precipitation products. Another promising application of the approach is the assessment of NWP models. More insights can be learned from the evaluation of NWP models by focusing on the local-scale precipitation structure.