UC Berkeley

AmeriFlux is a community of scientists measuring ecosystem carbon, water, and energy fluxes across the Americas with eddy covariance techniques. The network’s data team collects flux data for quality assessment and provides standardized data products to the earth science research community. Critical for scientists’ use of the flux data are the supporting Biological, Ancillary, Disturbance and Metadata (BADM) that provide context, such as measurement heights, instrument operations, and disturbance events. Managing and collating BADM into standardized data products are challenging due to their inherent long-tail data characteristics, i.e., they are diverse, free-formed, and infrequently measured. Over the past 12 years, we have worked with the community to standardize and then manage BADM using a SQL database with strong data quality criteria. BADM’s inherent nature demands rigorous quality checks of submitted data. Some of these checks provide feedback to data providers for correction, while others identify unusual cases necessitating dialogue with data providers. As we continue to gain experience with BADM, we collate and abstract similarities among data quality issues to build new automated checks and standards. Handling long-tail data presents interesting challenges and requires dealing with atypical “corner” cases. Our talk will describe our experiences in standardizing and managing BADM, and talk to the evolution of the system as we look towards better serving the earth science community.