Emergency Medicine Influencers’ Twitter Use During the COVID-19 Pandemic: A Mixed-methods Analysis

Introduction The objective of this study was to analyze the messages of influential emergency medicine (EM) Twitter users in the United States (US) during the early stages of the coronavirus disease 2019 (COVID-19) global pandemic by characterizing the themes, emotional tones, temporal viewpoints, and depth of engagement with the tweets. Methods We performed a retrospective mixed-methods analysis of publicly available Twitter data derived from the publicly available “Coronavirus Tweet IDs” dataset, March 3, 2020–May 1, 2020. Original tweets and modified retweets in the dataset by 50 influential EM Twitter users in the US were analyzed using linguistic software to report the emotional tone and temporal viewpoint. We qualitatively analyzed a 25% random subsample and report themes. Results There were 1315 tweets available in the dataset from 36/50 influential EM Twitter users in the US. The majority of tweets were either positive (455/1315, 34.6%) or neutral (407/1315, 31%) in tone and focused on the present (1009/1315, 76.7%). Qualitative analysis identified six distinct themes, with users most often sharing news or clinical information. Conclusions During the early weeks of the COVID-19 pandemic, influential EM Twitter users in the US delivered mainly positive or neutral messages, most often pertaining to news stories or information directly relating to patient care. The majority of these messages led to engagement by other users. This study underscores how EM influencers can leverage social media in public health outbreaks to bring attention to topics of importance.


INTRODUCTION
The newly discovered coronavirus disease 2019 (COVID-19) global pandemic has had an unprecedented impact on the healthcare community. As of this writing, there have been more than 73.7 million cases worldwide with the United States (US) accounting for approximately one quarter of all cases. The US also accounts for about one fifth of all deaths from COVID-19, with over 300,000 lives lost. 1 Since first being described in Wuhan, China, in December

Population Health Research Capsule
What do we already know about this issue? The use of Twitter by influential emergency medicine (EM) users during the coronavirus disease 2019 (COVID- 19) pandemic had not previously been studied.
What was the research question? What were influential EM Twitter users talking about during the early stages of the COVID-19 pandemic?
What was the major finding of the study? Influential EM Twitter users mainly shared news or clinical information in positive or neutral messages.

How does this improve population health?
This study shows how EM influencers used Twitter in a rapidly evolving situation, and may suggest how it could be leveraged in future public health crises.
Within the field of emergency medicine (EM), social media -in particular Twitter -has risen as a popular platform for the quick and widespread dissemination of information and opinions. 8 Opinion leaders within the EM community on Twitter have previously been identified as those with the most followers and most connections within the Twitter community. . 9 These EM influencers have a disproportionate impact on discourse due to their large audiences that view them as credible sources of information. 10,11 Prior to the COVID-19 pandemic, EM influencers used Twitter primarily to discuss medical topics and to share resources and opinions, with a significant social and humor component. 12 Since COVID arrived, doctors across many specialties have increasingly turned to Twitter to both gather information and to combat misinformation. [13][14][15] The goal of this study was to analyze the messages of influential EM Twitter users in the US during the early stages of the COVID-19 pandemic. We aimed to evaluate the thematic landscape of messages over time to help describe how social media was being used by the EM community in a novel and evolving setting. We also sought to analyze the emotional tone and temporal viewpoint of the language used and depth of engagement with these messages. These data provide insight into ways EM users can leverage social media in future health crises for the benefit of clinicians and patients alike.

Study Design
This was a retrospective analysis of publicly available Twitter data analyzed via a mixed-methods analysis using a combined content analysis approach. Due to the qualitative and quantitative nature of Twitter data, combined content analysis has been suggested to address these types of datasets. 16,17 Sample We analyzed a sample of English-language tweets from 50 influential, US-based EM influencers on Twitter during the early stages of the COVID-19 pandemic. We adapted the list of EM influencers from two previously published network analyses 9,18 using an iterative consensus-driven process; criteria included presence on either prior list, recognition within the EM Twitter community, and based in the US (Appendix 1). We excluded one potential EM influencer (NST), who is a coauthor on this study, to avoid coding bias since tweets cannot be fully blinded. We chose to analyze influencers from the US to narrow our study to one area of the world where the disaster dynamics were occurring under one governance structure and country environment, enabling a more nuanced analysis of themes related to the US healthcare system, regional logistics, and clinical practices. Previous studies have shown that analyzing tweets from Twitter influencers provides a narrative of Twitter activity without needing to analyze all users; 12 thus, we felt that limiting our sample to influencers would still reflect the general conversation among all EM users on Twitter.
Given the evolving nature of the pandemic over time, we decided to analyze a sample of tweets from each week during the time period of March 3, 2020-May 1, 2020. These dates range from the week before the World Health Organization declared COVID-19 a pandemic to the most current dates available at the time of data retrieval. 19

Data Collection
We used the George Washington University Libraries Tweetsets data platform to access and filter the "Coronavirus Tweet IDs" dataset (version 5). The dataset consists of Tweet IDs "collected using the POST statuses/filter method of the Twitter Stream API, using the track parameter with the following keywords: #Coronavirus, #Coronaoutbreak, #COVID19." 20 Version 5 of the dataset contains tweet IDs from March 3, 2020-May 1, 2020. The Tweetsets search functionality allowed us to generate a dataset from the Coronavirus Tweet IDs dataset (which contained 188,026,475 tweets). To generate the dataset for this investigation, we included only original tweets and quote tweets authored by the pre-specified list of EM influencers. Unmodified retweets and replies were excluded.
The Twitter developer policy 21 states that tweet IDs may be publicly shared for academic purposes; however, tweets may not. The dataset above contains only tweet IDs, not the actual tweets. Subsequently, tweet IDs were "hydrated" back to full tweets for purposes of analysis using the "Hydrator" program available at Documenting the Now (https://www. docnow.io). 22 "Hydrating" a tweet ID converts each numeric identifier into a line of data in a comma-separated values (csv) file that contains both the text of the actual tweet as well as additional metrics (eg, likes, retweets, location of the user, date and time of tweet, and URL links).

Analysis
We used a mixed-methods analysis with quantitative analysis performed on the full dataset and qualitative analysis performed on a subsample of the data.

Quantitative Analysis
Descriptive statistics are reported for the type of tweet (original content vs a retweet with comment, where a user comments on another tweet embedded within their tweet). The number of tweets per week is reported. The first week of data is only a partial week since March 3 was midweek (Tuesday). We described the reach and engagement of tweets using the number of followers of the EM influencers and the number of "likes" and retweets received. Twitter metrics, including retweets, mentions, and followers are considered traditional metrics of influence. 23 We analyzed the emotional tone of the tweets and the temporal focus using a linguistics approach. The Linguistic Inquiry and Word Count (LIWC) program developed by Pennebaker and colleagues has been used in previous medical and public health literature to evaluate linguistics in social media during acute crises. 24,25 Pronounced "luke," LIWC is a text analysis software with a predefined dictionary composed of 90 word categories with 6400 words and word stems that has acceptable prior validity evidence. These words and word stems reflect a variety of emotions, thinking styles, social concerns, and parts of speech. The LIWC output reports the ratio of the words in each category relative to the total word count of the analyzed text. 26,27 We used five categories from the existing LIWC dictionary in the analysis. To describe the emotional tone of the tweet we used the "positive emotion" and "negative emotion" categories (with sub-categories of "anxiety," "anger," and "sadness"). If the text of the tweet did not contain any words in the positive or negative emotion word categories, it was categorized as a neutral tone.
To describe the temporal focus of the tweet, we used the "past focus," "present focus," and "future focus" categories. If any words in the respective word categories were present, the tweet was categorized in that group. Groups were not mutually exclusive (eg, a tweet could express both positive emotion and negative emotion or have both a past and present focus). Descriptive statistics were used to report the quantitative metrics. All analyzes were performed using Stata 13.1 (StataCorp, College Station, TX).

Qualitative Analysis
Tweet text was analyzed inductively, following an emergent content analysis approach to allow for restructuring of coding categories if new themes emerged during analysis. 28,29 The coding team was comprised of four authors (DM, ML, MS, MM), all with different experiences and backgrounds in EM Twitter. The lead author (DM) is an emergency physician (EP) with extensive experience in qualitative research and rare use of Twitter. One author (ML) is an emergency medicine resident with no experience on Twitter. Two authors (MS and MM) are undergraduate students, one (MS) with extensive experience on Twitter, although no interaction with medical Twitter, and the other (MM) with minimal experience on Twitter.
The coding team read through a random sample of 50 tweets to develop initial coding categories using an inductive approach. We used a random number generator (https://www. random.org/integer-sets/) to create our sample sets. Tweets were viewed in a web browser for coding rather than reading the text alone (in csv file) to most closely approximate the viewing experience of the original audience and to allow for the added context of images. Quote tweets were coded based on both the content of the new text and the link or text being shared as often the quote text alone would have been insufficient to categorize the tweet (eg, "check this out"). In the case that the quoted content was no longer available on Twitter, the tweet was categorized by the quote text alone.
We refined coding categories in an iterative manner, and created a coding dictionary with definitions and sample tweets to serve as unambiguous examples. The initial coding categories and codes for the development sample were reviewed by one of the paper's authors (NST), an EM influencer whose tweets were excluded from this study to ensure that tweets were not misinterpreted or codes overlooked (member checking). The sample of 50 tweets used for code development was included in the final analysis.
Given the large size of the dataset, the qualitative analysis started with a goal of analyzing a 25% random sample of tweets from each week, with a plan to expand analysis to a 33% sample (and beyond) if there were new qualitative categories arising in the late stages of coding of the 25% sample (eg, if data saturation was not achieved). The random number generator was applied to each one-week time frame (rather than to the whole study period) to ensure balance across weeks because the authors suspected the topics covered on Twitter might vary week to week as different aspects of the pandemic evolved over time (eg, testing; personal protective equipment [PPE]). Previous studies evaluating Twitter content analysis within healthcare have analyzed between 288 and 1583 tweets. 12,30,31 After initial code development, three authors (ML, MS, MM) coded the remaining tweets with each tweet being double coded by a dyad of coders (eg, ML+MS, MS+MM, ML+MM). The full coding team met iteratively to discuss and reconcile any coding disagreements, revise the codebook, and develop new categories as needed. Strategies used to strengthen the validity and credibility of the data included member checking, memoing, reflexivity, and triangulation of data.

RESULTS
We identified a total of 1315 tweets from the 50 EM influencer tweets and quote tweets for the study period. The distribution of tweets across weeks is displayed in Figure 1. Tweets were split almost evenly between original tweets and quote tweets.

DISCUSSION
While previous studies have addressed the use of Twitter by physicians during the COVID-19 pandemic, 32 this is the first study to evaluate the use of Twitter specifically by EPs. The use of Twitter by EPs in public health crises is not new 33 ; however, the challenges of the COVID-19 pandemic create a unique backdrop in which to analyze Twitter data. Further, the online medical Twitter community provided a unique opportunity for EPs to share clinical information and experience, as well as personal stories and support, during a historic and rapidly changing global health crisis. Our results show a range of themes among the messages, most often related to sharing facts, local news, or information pertaining to clinical practice. There was a significant aspect of social engagement between users\ via likes and retweets, enhancing previous work that describes connections on Twitter as a network for collaboration and information sharing .9,17,18  Table 1). The most frequent hashtags appearing in the included sample are shown in Table 2; #covid19 was present in 1107 tweets, or 72.5% of the sample.
The qualitative dataset included 381 tweets (50 from derivation, plus the 25% sample of the remaining 1265). Four (1%) of the modified retweets did not have the original quoted tweet's text available, so were coded based on the author's quote text alone. Analysis identified six thematic categories encompassing 19 descriptive codes. These themes with exemplary tweets are shown in Table 4.
Just over one third of the tweets (131/381, 34.4%) shared facts or links to news outlets discussing testing, case volume, or other local stories pertaining to the pandemic. Over a

Metric n (%)
Describing the influencers
Similar to the pediatric intensive care Twitter community during the COVID-19 pandemic, as evaluated by Kudchadkar and Carroll, our study showed that EPs used Twitter to rapidly disseminate information about clinical practices as they continued to evolve. 34 Twitter as a platform inherently lends itself to this type of collaboration. With a median of 35,574 followers per account, influential EM users have a farreaching audience. Twitter thus can be a critical tool in helping EPs build their clinical framework for COVID-19 patients in a collaborative, dynamic environment. Particularly at the beginning of the pandemic when clinical trials and other more rigorous research were rare, sharing personal experiences and clinical information may have helped shape clinical practice and care protocols. EM Twitter messages in this sample have more positive or neutral emotion words in comparison to messages by the general public on Twitter. 35,36 This is a notable finding since the healthcare system and frontline workers were and continue to be among the most negatively impacted by the pandemic. 37,38 It points to the importance of different perspectives in shaping attitudes and sentiment. This disparity in sentiment between specific populations is worthy of future investigation, not only within the EM community but within the larger field of crisis informatics.
Similarly notable, unlike Rufai and Bunce's evaluation of G7 world leaders on Twitter in the early pandemic, 39 our analysis did not identify morale-boosting messages as a significant theme. At the beginning of the pandemic, healthcare professionals were likely more focused on clinical practice and overwhelmed healthcare systems. The role of the user offline, both in a personal and professional capacity, likely relates to the role a user takes on Twitter and may account for these differences.
Studies of Twitter data during previous public health outbreaks have suggested roles for the social media platform during dynamic and uncertain times like the COVID-19 pandemic. These roles include infectious disease surveillance, predicting spread, dissemination of public health information, and assessing public views of the outbreak. [40][41][42][43][44] Twitter messages sent during previous emergency and mass convergence events reveal features of information dissemination that support information broadcasting and brokerage. 45,46 For example, during Hurricane Isaac in 2012, public health situational awareness in non-traditional format was shared through Twitter .47 Specialized groups are often part of trusted networks that are crucial during disasters since they form a network of individuals and groups that either formally or informally pursue a common goal or purpose. 48 Information exchange within these groups is often perceived as not only more credible but often more relevant. The EM influencers and their tweets represented in this analysis likely represent a medical specialty-focused trusted network with wide reach both within EM and to the general public.
There is also a role specifically for public health officials and physicians to combat misinformation on traditional and social media. 49 Misinformation on Twitter during the COVID-19 pandemic is already prevalent and negatively impacts public perception of the virus and can inhibit adherence to public health initiatives. 50,51 While we did not evaluate the scientific accuracy of any of the tweets, many tweets identified and addressed perceived misinformation. This is congruent with the findings of Wahbeh et al that physicians across specialties have been using Twitter to warn the general public about misinformation relating to COVID-19. 32 Some have described the need for a concerted effort to train healthcare professionals and the general public in appropriately evaluating social media as a result of the widespread use of social media during the COVID-19 pandemic. 52 Future work may evaluate the presence of misinformation in medical tweets during the pandemic and specifically explore the role of physicians in combatting misinformation during the pandemic.     Our results demonstrate that influential EPs on Twitter are participating in conversations surrounding the COVID-19 pandemic to further clinical practice, spread information, and relay personal experiences and opinions. They are using mostly positive or neutral language, although not in a way that is seen as morale-boosting. These results may provide guidelines and help enable and encourage EM Twitter users, particularly those who are influential, to use Twitter to advance clinical care, increase public awareness, and promote health initiatives.

LIMITATIONS
We analyzed content from US users only. Similarly, we limited our sample to English-language tweets. The EM Twitter community is international and multicultural, and findings may not be generalizable to this global network. 9, 18 We chose to focus solely on EPs instead of including other specialties. This choice may have led to missing significant themes and messages among the larger medical community on Twitter. Further, although we updated the list of EM influencers, it was originally formulated in 2015 and may be out of date or may not accurately represent current drivers of discourse in the EM Twitter community. This reflects a lack of a standardized method of identifying these influencers in the literature and may warrant the development of a systematic approach of identification of users for future research. We were also using Twitter as a surrogate for social media platforms as a whole. This focus on Twitter may have excluded discussions and themes unique to other platforms such as Facebook or Instagram.
Our coding team did not include any avid EM Twitter users. While this choice lent a more neutral lens to the data analysis, it may mean that nuances of the EM Twitter community were not captured in the analysis. As with all qualitative studies, there is possible inherent bias due to coding by individuals. To combat this potential bias, we used a large coding team comprised of individuals with multiple backgrounds.
The main dataset itself is also a limitation of this study. As inclusion in the dataset was based on a narrow set of hashtags, certain themes may have been missed if those exact tags were not included and tweets with hashtags may not be representative of tweets in general. For example, the #getmePPE movement made many headlines in traditional and social media but was present in only 2.2% of our data set. 53,54 This low rate of #getmePPE may be because tweet authors didn't routinely include #COVID19 or the other inclusion criteria in their #getmePPE messages. A small number of tweets no longer had the quoted content available, which may have led the coders to mis-categorize the tweet. This lack of quote content, however, was a rare occurrence and likely did not significantly affect overall percentages of tweets in each theme. Lastly, the pandemic is ongoing, and conversations are Leibowitz et al.
EM Influencers Twitter Use During COVID-19 ever evolving; themes and emotional content we identified may no longer be as prevalent.

CONCLUSION
During the early weeks of the novel coronavirus pandemic, influential emergency medicine Twitter users in the United States delivered mainly positive or neutral messages, most often pertaining to news stories or information directly impacting patient care. The majority of these messages led to engagement by other users in the form of likes and retweets. This study underscores how EM Twitter influencers can leverage social media in public health outbreaks to bring attention to topics of importance.