Lawrence Berkeley National Laboratory

Well-worn metrics like Journal Impact Factor (JIF) as well as new, complex indicators like the CDt index all share the assumption that all citations are created equal. However, due to a lack of standards in citation practices as well as the inherent complexity of academic discourse, we cannot expect this assumption to hold. One citation may indicate strong and pervasive influence of an older article on a new study, while another citation of that same older article may represent nothing more than a cursory mention, the inclusion of which does little to influence the claims made by the citing article. Professionals attempting to use citations as a tool for understanding the impact of their organizations are thus handicapped by the caveat that a large body of citations will contain examples representing widely varying degrees of actual, tangible impact. My talk will cover recent efforts at the US DOE Joint Genome Institute (JGI), part of the Lawrence Berkeley National Laboratory, to begin accommodating for the realization that not all citations are created equal. I will discuss a two-pronged approach, the first component of which involves capturing all possible citations of JGI's research outputs. Citations of these outputs flow through a wide variety of metadata, from publications to URLs to persistent identifiers. If we want to understand how JGI influences the larger scientific community, we need to first capture all identifiable JGI citations regardless of the product or type of product being cited. The second prong of our approach involves the binning of citations according to the level of JGI involvement indicated by a given citation. Using results from a combination of comparative searches and manual evaluation based on random samples, I propose a 'spectrum of JGI involvement' for citations of two large-scale JGI initiatives. Preliminary investigation of these results indicates that the majority of citations we can identify represent a relatively weak influence of JGI upon the citing articles, while instances of stronger, more tangible impact are relatively few and far between. The implications of these preliminary results are broad and apply to research assessment professionals across a wide range of organizations. I will begin with a description of the problem as it applies to JGI, followed by methods and results. I will then conclude with a discussion of how this work fits into existing community initiatives and describe opportunities for future development.