Documentation of Screening for Firearm Access by Healthcare Providers in the Veterans Healthcare System: A Retrospective Study

Introduction Presence of a firearm is associated with increased risk of violence and suicide. United States military veterans are at disproportionate risk of suicide. Routine healthcare provider screening of firearm access may prompt counseling on safe storage and handling of firearms. The objective of this study was to determine the frequency with which Veterans Health Administration (VHA) healthcare providers document firearm access in electronic health record (EHR) clinical notes, and whether this varied by patient characteristics. Methods The study sample is a post-9–11 cohort of veterans in their first year of VHA care, with at least one outpatient care visit between 2012–2017 (N = 762,953). Demographic data, veteran military service characteristics, and clinical comorbidities were obtained from VHA EHR. We extracted clinical notes for outpatient visits to primary, urgent, or emergency clinics (total 105,316,004). Natural language processing and machine learning (ML) approaches were used to identify documentation of firearm access. A taxonomy of firearm terms was identified and manually annotated with text anchored by these terms, and then trained the ML algorithm. The random-forest algorithm achieved 81.9% accuracy in identifying documentation of firearm access. Results The proportion of patients with EHR-documented access to one or more firearms during their first year of care in the VHA was relatively low and varied by patient characteristics. Men had significantly higher documentation of firearms than women (9.8% vs 7.1%; P < .001) and veterans >50 years old had the lowest (6.5%). Among veterans with any firearm term present, only 24.4% were classified as positive for access to a firearm (24.7% of men and 20.9% of women). Conclusion Natural language processing can identify documentation of access to firearms in clinical notes with acceptable accuracy, but there is a need for investigation into facilitators and barriers for providers and veterans to improve a systemwide process of firearm access screening. Screening, regardless of race/ethnicity, gender, and age, provides additional opportunities to protect veterans from self-harm and violence.


INTRODUCTION
In 2020 42% of United States (US) households reported owning a firearm. 1 Firearms in the home increase risk of violent events, [2][3][4][5] and is a significant threat to public health. Nearly half (44.9%) of all US military veterans own a firearm, with ownership reportedly higher among males (47.2%). 6 Veterans are at disproportionate risk for suicide, 7 accounting for 20% of suicide deaths despite constituting 13% of the US population. Firearms are involved in 67% of suicides among veterans compared with 50% of the general public. 8 While access to firearms is associated with increased risk for injury and death, safe firearm storage is associated with decreased risk. 9,10 Public health advocates recommend strategies to restrict access to lethal means as a suicide prevention strategy. 11 For firearms these processes include safe storage measures such as gun safes, gun locks, storage of ammunition and guns separately, and storage of guns unloaded and locked. 12 Members of the military tend to store firearms unsafely with 45.2% reporting they store firearms both loaded and unlocked, and an additional 33% store firearms either loaded or unlocked. 13,14 Screening veterans for firearms ownership and safe storage is needed to prevent unnecessary injuries and deaths. Healthcare providers are in a position to screen and counsel patients on safe firearm storage. 4 Counseling on health and safety is a well-established healthcare practice; there are guidelines for screening and counseling in many areas of health including healthy eating, physical activity, mental health, and injury prevention. 15 While firearm-related injuries and deaths are a public health problem, particularly in the US, 16 a minority of physicians report engaging in firearm counseling. 17 Despite several groups having recommended both targeted and universal screening for firearm access, [17][18][19][20][21][22][23][24] there are no current national guidelines for screening in primary care, urgent care or emergency care settings even though gun safety is associated with lower risk of injuries and death. 25,26 To understand how current practice may be adapted, more information on the frequency with which healthcare providers document firearm screening is needed. In this study we present results of Veterans Health Administration (VHA) healthcare providers' documentation of firearm access screening in electronic health record (EHR) notes among VHA patients in outpatient primary care, urgent care, and emergency department (ED) settings.

METHODS
The study is a cross-sectional examination of the frequency of documentation of screening for veterans' access to firearms across several healthcare settings using natural language processing (NLP), which refers to automatic computational processing of human language. 27 The study was approved by the Veterans Administration Connecticut Healthcare System Institutional Review Board.
The study sample included men and women veterans from a national, post-9-11 cohort 28,29 during their first year of VHA healthcare, defined by the presence of at least one primary care visit from 2012-2017. We obtained data on demographic and veteran military service characteristics from the Defense Manpower Data Center-Contingency Tracking System Deployment File, provided to the VHA from the US Department of Defense. Variables included age, gender, race/ethnicity, marital status, education, rank (e.g., officer, enlisted), military branch (e.g., Army, Marine Corps), and deployment dates. VHA visit information came from EHR data extracted from the Corporate Data Warehouse (CDW). The CDW includes information on healthcare utilization, pharmacy, laboratory, vital signs, coded diagnostic and procedural data (International Classification of Diseases, 9th and 10th revisions, Clinical Modification [ICD-9-CM and ICD-10-CM]) and Current Procedural Terminology (associated with all VHA inpatient and outpatient encounters. 30,31 We identified comorbid conditions using ICD-9 and ICD-10 coded diagnoses defined by ≥2 outpatient (on separate days) or ≥1 inpatient code for the condition. This methodology has been used for the identification of psychiatric disorders in administrative data 32 and human immunodeficiency virus in Medicaid data. 33 Diagnostic code groupings were previously validated. 34 Major mental health diagnoses included posttraumatic stress disorders (PTSD), major depressive disorders,

Brandt et al.
Screening for Firearm Access by Healthcare Providers in the VA System alcohol use disorders, and substance use disorders.

Natural Langauge Processing Tool Development and Performance for Firearm Access Identification Firearm Taxonomy
For the information extraction process, we developed a coding manual for chart review and a taxonomy for firearms for annotation. A taxonomy was created by by searching existing vocabularies (NCBIO, UMLS, SnoMed) and the literature for published ontologies used for guns, gunlock, and firearms. A Cochrane review on gunshot wounds contained terms such as trauma* or injur* or penetrat* or wound*or perforat* or stab* or gunshot or shot, and the Medical Subject Headings (MeSH) database included the following: "Wounds, Gunshot"[Mesh]) OR "Wounds, Penetrating"[Mesh:NoExp]) OR"Multiple Trauma"[Mesh])) OR "polytrauma." This list of terms was supplemented with over 120 candidate terms and phrases contained in a national database 35 on firearm homicides. We then reviewed and narrowed the phrases down to 27 (shown in Table 1) deemed relevant by VHA clinicians.

Annotation
We identified 2,584,607 notes with one or more of the phrases, and annotated 1856 text snippets randomly selected from notes that contained any of the search terms. Each snippet contains a 35-word span before and after a firearm-relevant phrase. The annotation classifications for firearm access were the following: positive (ability to determine that the veteran had current access to at least one firearm); negative (language that the veteran did not have current access to any firearms); and ambiguous (there was insufficient evidence for either a positive or negative classification from the note -an example might be that the veteran owned a firearm but it was somewhere else). Each snippet was annotated by two of the authors and disagreement adjudicated by their consensus. An inter-annotator agreement was calculated. The annotated snippets served as the reference standard in training and testing.

Features
We used n-grams as features. In clinical text, unigrams are single words, and bigrams are two words that occur in a sequence. For example, in the phrase "patient owns a shotgun" the unique unigrams are patient, owns, a, and shotgun. In the same phrase, patient_owns, owns_a, a_shotgun are unique bigrams. Alpha or numeric tokens (discrete words and numbers) were counted in the unigrams and bigrams. The features included unique unigrams with a frequency greater than 34, and unique bigrams in the annotation spans with a frequency greater than four. These threshholds are empirically chosen to filter out the less prevalent n-grams and reduce overfitting. The training features for the model (for each document) consisted of binary indications of the presence of each of the identified unigrams and bigrams, along with the offset location of the keyphrase in the snippet.

Training and Testing
We used the annotated snippets to train a random forest model with 200 estimators or trees. The random forest model maximum depth was set to 15, with maximum features automatically determined by the model and the gini split criterion. Hyperparameters were determined through gridsearch and other testing. We split the 1856 text snippets into 85% for training and 15% for testing. The model performance was measured by accuracy.

Validation
For validation, we annotated an additional 238 clinical notes on the note (instead of the snippet) level, with 175 negtive and 63 positive for firearm access. The random forest model was applied to these notes, based on the snippet identified in each document. Figure 1 below is a schematic of this process.

Statistical Analysis
We conducted all statistical analyses using SAS software version 9.4 (SAS Institute, Cary, NC). Baseline characteristics of veterans include frequency (percentages) and means (± standard deviations) or median (interquartile range), and differences by age, race, ethnicity and gender were examined using chi-squared test or Student's t test, as appropriate. We used a multivariable logistic regression model to assess firearm mention and adjust for potential confounding based on the literature. Among those with firearm mention, a logistic model was run to assess firearm access. We examined model fit using quasi-information criterion and residual plots. Hypotheses were tested at a two-sided significance level of α = 0.05. Screening for Firearm Access by Healthcare Providers in the VA System Brandt et al.

RESULTS
The Cohen kappa score measuring inter-annotator agreement among the review team members identifying screening documentation incidents was 80%. On the testing dataset (15%), the accuracy was 81.0%. On the final validation dataset, the random forest model achieved 81.9% accuracy, 90.9% specificity, 57.1% sensitivity, and positive predictive value of 69.2% in classifying the 238 test notes. Table 1 demonstrates the frequency of the most common firearmrelated terms within the VHA text notes. These counts are non-distinct by patient but demonstrate the breadth of terms used in clinical notes by providers; many of the highly specific terms were present in notes as historical and exposure events for PTSD documentation, and/or noise exposure (out of 105,316,004 outpatient care notes).
We included data during the first year in VHA care for 762,953 veterans in the analytic sample. Table 2 demonstrates the frequency of documentation of access to firearms and other guns by clinicians within one year of entry into VHA healthcare. The mention of any firearm within a clinical note for veterans was 9.8% of men, 7.1% of women, and 6.5% in veterans over 50 years of age. Among the small number of veterans with any firearm term present, only 24% were classified as positive for access to a firearm (24.7% of men and 20.9% of women [data not otherwise shown]). Prevalence patterns by race of any mention/positive access were similar, with the highest rates among Whites (9.6% mention and 26.3% access [data not otherwise shown]). Documentation of firearms was higher in veterans with higher numbers of mental health visits, emergency and urgent care visits than primary care. Documentation of firearms did not vary regardless of the number of primary care visits (data not shown). After adjustment for demographics, utilization, and comorbidities, significant differences in documentation of access remained by age, gender, and among veterans with major depression or PTSD diagnoses.

DISCUSSION
Results demonstrate documentation of firearm access in clinical notes for less than 10% of contemporary veterans within the first year of enrollment in VHA healthcare, and that nearly one quarter of those with documentation were identified as having access to a firearm. There was a significantly lower rate of documented access for women veterans, despite data that show high rates of both men and women veterans who live in homes with firearms, and increasing rates of fiream-related suicides among women veterans. 6,36 While documentation does not always equate with conversations between providers and patients, the low frequency of documented patient-provider interactions seen in this population suggests that there is a clear opportunity to increase initiation of conversations about firearm access and safety. Barriers to implementation of firearm screening and safety counseling include provider uncertainty about the effectiveness of firearm screening, provider uncertainty about the legality of asking about firearm ownership, and provider unfamiliarity with firearms. Further, provider unfamiliarity with lethal means restriction as a firearm suicide prevention strategy may prohibit uptake of screening and counseling. 17,23,[37][38][39] These barriers indicate a need for increased training of healthcare providers on firearm screening and safety counseling and normalizing the opportunities to discuss firearms in a population that has higher rates of firearm ownership and use.
Discussions must be acceptable to providers and to patients for it to be effective. Roszko and colleagues' review of 53 studies of non-veteran clinician firearm attitudes and practices found that positive attitudes toward firearm discussions were higher than actual documented discussions, with low firearm discussions across all disciplines. 17 This is encouraging in that it could indicate healthcare providers may be willing to undergo training in initiating and carrying out these discussions, although it remains to be seen whether attitudes differ among VA providers.
While providers may have positive attitudes toward firearm screening and counseling, recent studies show mixed support by gun owners and veterans for healthcare provider initiation of gun safety conversations. 11,40 This suggests that while providers may be willing to initiate these discussions, it is not clear that patients will welcome or participate in them if initiated. Such conversations will need to be clearly delineated as prevention oriented for gun-owning citiziens and families with specific, evidence-based practices such as the following: Homes with locked guns are less likely to have unintentional or self-inflicted injuries with firearms or deaths. 41   Perhaps related to the reasons specified above, the evidence for the effectiveness of this firearm safety conversation in the clinical setting is mixed. 42,43 For this reason, appropriate, acceptable communication must be used and evaluated to maximize the impact and inform the knowledge base of these efforts in the clinical setting. 44,45 However, a recent epidemiologic review indicates that counseling combined with safety-device provision can impact safe storage in the community. 41 Promising strategies include following the guiding principles of shared decision-making, with providers stating neutral risks and protective factors related to gun safety and involving gun owners in the development of messaging. 46,47 These neutral risks can be culled from the US Centers for Disease Control and Prevention data as simple facts for patients to understand and acknowledge as part of their gun ownership responsibilities. 48 Specific to the VHA, an appropriate clinical response to the public health problem of firearm suicide in the veteran population is needed. Further research within the VHA is needed to determine the healthcare setting(s) and provider types most appropriate for firearm screening and counseling interventions. This step will require a participatory approach among health services and informatics researchers to improve the feasibility, acceptability, relevance, and sustainability of interventions. [49][50][51][52] In addition, research is needed to determine the modality and intervention format (electronic, face to face, written) that are most effective for each of the key domains in firearm injury research. Data on the moderators of acceptability and effectiveness (demographics, political views, comorbidities, etc.) of screening and interventions from the veteran and provider perspectives are needed. Only then can researchers begin to measure the short-and longer-term outcomes of such interventions and policies. While this approach is clearly specific to the clinical context and persons involved for veteran prevention with firearms, prevention is likely best on the frontlines of care and where repeated encounters occur with trust-building relationships. Thus, primary care, mental health and ED settings/providers may need to partner with the health services and health informatics researchers to fully address the scope of this need and develop interventions that fit the veteran patients and the VHA system. Equipped with information and curiosity, clinicians can engage their veteran patients as part of routine care, instead of urgent or emergent care, and the health services and health informatics teams can inform us about which methods are most feasible and impactful for veteran quality of life and provider use and sustainability.

LIMITATIONS
Given retrospective studies may introduce sampling bias 53 , we included the entire population, not a sample. The results of the NLP algorithm were limited for the first year of entry into VHA healthcare for years 2012-2017, which might underestimate firearm documentation. The identification of firearm documentation for patterns such as temporal changes, variations in types of providers and provider settings, and other patient characteristics will be explored in future work. For example, in this sample there were increases by year (from 3% in 2012 to 21% in 2017). Further research is needed to help explain this increase.

CONCLUSION
Natural language processing methods are able to determine the prevalence of documented firearm screening and safety counseling across a large population of US military veterans. We identified low prevalence of firearm access screening documentation and believe that further investigation into facilitators and barriers is necessary. This work should inform the process for development of systemwide practices to reduce firearm suicide and injury among US veterans, a large group at elevated risk.