Evaluation of Online Training on the Prevention of Venous Thromboembolism

Introduction: The integration of new evidence into clinical practice can be a prolonged process, with delays of years or even decades. One approach to speed this integration is through the use of online provider education. Problem: Venous thromboembolism (VTE) is a serious patient safety issue. Prevention requires coordinated care and adherence to evidence-based guidelines, supported by provider education. Purpose: This study reports how an interdisciplinary team developed and piloted an online provider training program for the prevention of VTE. Hypothesis: If providers use the online educational training, they will demonstrate increased mastery of key content areas related to VTE prophylaxis. Methods: We used a prospective test—retest study design in which medical residents and fellows served as their own controls. All participants were given a pretest followed by educational content and then a posttest. We also assessed 2 different types of learning content (ie, with and without case studies/questions) and randomized participants to each type prior to assessment. Results: Using the McNemar test we found a trend for knowledge gains related to VTE guidelines on the posttest for clinicians (n = 67) with a 14.5% improvement in content mastery (P = .05, 2-tailed). We did not find any significant differences between training modalities. Clinicians overall reported high levels of satisfaction with the application. Conclusion: Our online education efforts indicate the potential for increasing mastery of VTE prophylaxis concepts. If resources are limited, we suggest a static approach to content delivery and an exploration of standardized methods for portability of online curriculums across learning management systems.


Introduction
Venous thromboembolism (VTE) is a significant patient safety concern. Venous thromboembolism may manifest as deep-vein thrombosis (DVT) or pulmonary embolism (PE), with the latter estimated to be the most common cause of hospital death. 1 Each year approximately 100 000 deaths are attributable to VTE in the United States. 2 The management of VTE requires coordination of care across multiple health care providers supported by a robust system of care. 3 Despite guidelines and incentive programs offered by public and private entities for VTE prophylaxis, prophylaxis methods are still underutilized. 4 This reflects the frequently cited fact that when new evidence appears in the literature, it can take up to 17 years to make it into practice. 5 One approach to moving evidence into practice more quickly is through the use of online provider education. 6 This is particularly true for high-risk clinical problems in which compliance with rapidly changing guidelines is essential to improve patient care and maintain patient safety, such as VTE prevention.
The benefits of online education are numerous: learners can access content at a place and time that is convenient for them; administrators have the ability to track completion rates; usage of electronic order sets can be simulated; integration of multimedia content can be provided to supplement otherwise static content; and content can be updated in a distributed and collaborative fashion by experts in the field. The needs of practicing health care professionals, in light of time and schedules, make online education particularly compelling. 7 The purpose of this report is to describe how an interdisciplinary team representing 2 academic medical centers developed and piloted an online training program for the prevention of VTE. We report the results of a prospective trial in which we examined the effect of our online intervention on knowledge gains. This effort was part of a larger patient safety study supported by the Agency for the Health care Research and Quality Partnerships in Patient Safety that resulted in the development of clinical guidelines for the prevention, diagnosis, and management of VTE (http://vte.son.washington.edu/). 3 The interdisciplinary team of experts, representing nursing, medicine, pharmacy, and public health, recognized that provider education was an important component of implementing the VTE Prevention Guidelines. We wanted to develop training that was interactive, engaging, relevant, and measurable.
Several internal pressures complimented our efforts: Venous thromboembolism prevention was part of the joint medical center's strategic plan and was included in the operating budget. In addition, the medical center had just adopted a new learning management tool with a plan to rollout online training for multiple patient safety concerns, and wanted to know how best to engage the learner. The current way of ''pushing'' mandatory training, such as Health Insurance Portability and Accountability Act (HIPPA) training, was using static information with a test at the end of each section. Leaders in the medical centers wanted to test a provider's knowledge before and after specialized content were reviewed so that improvement could be ascertained. They also wanted to see whether an interactive ''question/ response'' mode would help reinforce the static content. These needs helped guide our research design.

Hypothesis and Research Questions
The working hypothesis of this research effort was that if providers use the online educational training, they will demonstrate increased mastery of key content areas related to VTE prophylaxis on the posttest compared to the pretest. A secondary hypothesis concerned the educational design. We posited that reinforcing the static lecture slides with real-life question/response case study questions would provide an opportunity for immediate feedback and a more robust learning experience, resulting in increased mastery of key points versus less mastery for those receiving only static lecture slides. We were also interested in a number of research questions related to how satisfied participants were with their learning experience and how participants used the application.

Methods
We used a prospective test-retest design to assess the effects of our educational intervention. Residents and fellows (n ¼ 69) from both medical centers consented to take part in the study, providing pretest scores, completing the educational intervention, and providing posttest scores and responses to an internally developed satisfaction scale. We also randomized participants between educational designs to answer our secondary hypothesis. This study was approved by the university institutional review board.
The training was promoted with 2 separate e-mails sent by the associate medical director of one of the medical centers and the surgical residency director. The e-mail carried the names of the medical directors of both medical centers. The first letter was sent in early September 2007 and the second was sent approximately 6 weeks later. Cross-links were also placed in various intranet pages including the medical centers' ''Clinical Toolkits'' page. Each computer desktop had a shortcut pointing to this Clinical Toolkits page.
The Web application was constructed so that the home page contained a message from 2 physician champions that explained the importance of VTE prevention (Appendix A). From this page, participants were able to log-in using a federated university username and password. Participants were then presented with an explanation of the research study with the option to consent or not (Appendix B). If they consented, a randomization procedure assigned them to 1 of 2 learning groups, the: (a) Usual Care (UC) group received the education as 9 ''static'' screens, with each containing a case scenario and the solution; the screens were embedded within the Web site using a slides metaphor (Appendix C), and (b) Enhanced Learning (EL) group who received the same content as well as between 4 and 16 question/ response case studies (Appendix D). If they chose not to participate in the research aspect of the online training, they received the same intervention as those in the intervention group, as we posited this was the best educational approach. Following the consent step, participants completed a brief demographic survey before undergoing the educational training and the posttest.
Our interdisciplinary team established a series of business rules to guide development efforts: (a) completing the demographic questions were optional, (b) participants would be presented with a 10-item pretest; and scores and incorrect answers would not be displayed. Both groups had the same pre-and post-knowledge tests administered. Each test had 7 true/false questions and 3 multiple choice questions dealing with medication dosing. All questions were similar in content and theme and developed by consensus based on recommendations from the 2004 American College of Chest Physicians VTE Prevention Guidelines 8 ; (c) all users would then be presented with 9 static ''slides'' typifying different clinical scenarios, (d) after viewing the slides, those in the EL group received a series of question/response case-study questions, these would be pulled from 4 blocks of interactive questions-each with 4 questions. One question would be chosen randomly from each block. If the participant failed that question, they would be given the reason why and then presented with another question from the same block. Failure of all 4 in a block redirected the participant back to the static slides; a correct answer led the participant to the next block of questions. At a minimum, a participant could answer 4 interactive case studies successfully and advance to the posttest. (e) Participants would be redirected to the training if they failed the posttest, however on the third failure they would be allowed to directly visit the posttest, (f) participants would receive feedback on the posttest for incorrect answers and those receiving a passing score of at least 80% would be able to print out a certificate of completion containing a unique code; administrators could verify that this code matched a participant in the database. Those with failing scores were provided with a link back to the beginning of the educational content. To help illustrate the effect of business rules on navigation paths, a wireframe for the Web application is presented below in Figure 1.
For development tools, we explored a number of authoring environments that allow nonprogrammers the ability to develop static content, as well as multimedia content and online quizzes. We were particularly interested in authoring environments that allowed course content to be exported using a standard-based format known as Structured Content Object Reference Model (SCORM) packages which could then be uploaded to learning management systems (LMS). 9 These systems are essentially sophisticated Web applications designed to track student progress in various learning modules. Modules authored with SCORM compliant tools can be uploaded into a compliant LMS and, as students progress through the learning materials, the module communicates critical information to the LMS, such as the name of the module, the student's progress within the module, quiz scores, and completion status. 10 The university where this research took place operated 2 different SCORM compliant LMSs (SumTotal and Moodle). However, we found that both lacked the features necessary for the research aspect of our study. Specifically, we were unable to build in randomization procedures necessary to branch participants to different learning pathways, and we could not collect the level of detail that we wanted on usage patterns.
As a compromise, we built our own Web application by leveraging blocks of code from past research projects designed for patient education. When possible, we built components, such as the didactic slides, using a SCORM compliant tool (Powerpoint with the Adobe Breeze Presenter plug-in) with the anticipation that following completion of the research study they could more easily be incorporated into a SCORMcompliant LMS.
For the application development, we used participatory design techniques including a needs assessment, analysis, and an iterative design/testing/development cycle as suggested by Mayhew. 11 Data from the online education training were exported from the Web-server database (MySQL) and imported into Statistical Package for the Social Sciences (SPSS) version 16 for Windows, 12 where it was examined for data irregularities prior to analysis.

Analysis
For the prospective randomized pilot, categorical variables were expressed as frequencies and percentages and continuous variables as means with standard deviations. In univariate analyses, the demographic characteristics were compared using the chisquare (w 2 ) tests or McNemar tests for categorical variables. Analysis of covariance (ANCOVA) was used to examine the effect of different types of learning modalities controlling for baseline knowledge. Hypotheses were tested using 2-sided tests, with P values < .05 considered to be statistically significant.

Setting and Participants
Physicians at 2 academic-affiliated medical centers were asked to complete the online provider education. Both academic centers were based in the same city in the Pacific Northwest, were affiliated with the same university, and were nonprofit public medical centers. As of 2009, the first center had 450 licensed beds, 1823 physicians, and saw over 19 000 admissions per year. The second center is a level 1 adult and pediatric trauma and burn care center with 413 licensed beds, 1216 physicians, and also saw over 19 000 admissions in 2009. Together, both centers and affiliated clinics saw over 1 million outpatient and emergency room visits. The distance between both centers is approximately 4 miles. The research participants were fellows and residents from both academic centers and represented all disciplines.

Results
A total of 89 participants accessed the system between September 6, 2007 and March 12, 2008 and consented to participate in the research study. In all, 20 participants failed to complete both a pre-and posttest. Some of these participants (n ¼ 10) never progressed beyond the login screen, the remainder did not complete the posttest. These participants were excluded from further analysis, given that no subsequent data were available.
Descriptive statistics for the remaining 69 participants (Table  1) are provided below. An examination of responses to the demographic questions for the excluded 20 participants who did not complete both a pre-and posttest found no telling patterns or trends. Of the remaining participants, slightly more than half (52.2%, n ¼ 36) were randomly assigned into the Usual Learning (UL) group and 47.8% (n ¼ 33) were assigned to the EL group. No significant differences in demographics were found across learning groups or in their pretest scores. All participants responded to the gender question with slightly more than half (55.1%, n ¼ 38) indicating that they were female; almost all of the participants (n ¼ 67) provided a response with respect to their discipline. Of these, the majority came from surgery (25.4%) or reported their discipline as ''other'' (25.4%). All participants provided a response with respect to age category, and the majority was in their 30s (56.5%).

Primary Hypothesis
Hypothesis 1: If providers use the online educational training, they will demonstrate mastery of key content areas related to VTE prophylaxis on the posttest compared to the pretest.
Overall, participants (n ¼ 69) had an average score of 79.28 (+12.17) on the baseline test. Results from the initial posttest found an increase from baseline with an average score of 82.32 (+13.84; Table 2), however, this increase was not statistically significant using a 2-tailed paired samples t test and an a of .05, (t ¼ À1.655, P ¼ .102), leading us to fail to reject our null hypothesis. The standardized mean difference effect size was 0.23, which may be considered a small effect, according to Cohen. 12 Analyzed with nonparametric methods, 47 (68.1%) participants displayed mastery at pretest with a score !80, while 57 (82.6%) displayed mastery at posttest (Table 2), representing a 14.5% improvement in mastery. Of the 47 masters at baseline, 6 regressed and became nonmasters on posttest, whereas among the 22 nonmasters at baseline, 6 stayed nonmaster and 16 demonstrated mastery. This dichotomous relationship was examined with the McNemar test (P ¼ .05).  An examination of individual items found questions related to dosing to be problematic for participants on the pretest. Less than 40% of participants answered multiple choice or true/false questions correctly about appropriate dosing for elderly patients as shown in Table 3. The third worse ranked question on the pretest concerned dosing for a middle-aged patient. The remaining questions all had more than 80% of participants respond correctly. As seen in Table 3, participants also had difficulty with dosing-related questions on the posttest, with 2 of the 3 worse-ranked questions involving appropriate dose questions in a multiple-choice format, however the frequencies of correct responses were much higher on these questions compared with similar questions asked on the pretest.

Secondary Hypothesis and Research Questions
Hypothesis 2: Providers who receive the question/response case studies (EL) after the static slides will score higher on posttest than students who only receive the static slides (UL).
Participants were randomized to 2 learning styles following consent. Participants in both groups scored relatively high on the pretest; the UL group (n ¼ 36) had a mean score of 80.28 (+13.2), while the EL group was slightly lower at 78.18 (+11.1). Posttest scores found a mean score of 85.0 (+12.3) for those in the UL group compared to a mean score of 79.39 (+15.0) for those in the EL group; there was no significant effect of learning modalities on the posttest controlling for the baseline knowledge in the ANCOVA (F ¼ 2.38, P ¼ .13).

Satisfaction
We were also interested in learning how satisfied participants were with their learning experience and how participants used the application. As indicated in Table 4, participants responded favorably to the training program indicating that it was helpful, that the time was acceptable, and that they were satisfied overall. There were no significant differences between the UL and the EL groups on any of these items.
A total of 14 free text comments were provided in response to ''What suggestions do you have for improving this site?'' One comment disputed an incorrect answer on the posttest, but the participant was unsure what answer selection they had given. Four comments indicated a dislike of the linked PDF guidelines that were available in both the passive and interactive didactic, an example comment was ''don't have people reading attached PDF's!! rather summarize the content in one of the slides of the course.'' These comments were countered by 1 participant who said ''[I] liked the printable handouts (pdf files).'' One comment said we should review the site for typos and 4 comments were positive including ''thanks for the case studies. Should be a good primer for folks,'' and ''Good format and links are helpful.''

Discussion
We would encourage health care organizations to consider offering this form of online training as part of comprehensive VTE prevention efforts. Although our effect size 13 may be classified as small at 0.23, this is an important area where changes in knowledge and attitude can prevent morbidity and mortality. Overall, we saw a 14.5% difference in mastery between pretest and posttest, and we feel this warrants further study. Our analysis of individual test items indicates that correct medication dosing should be emphasized within training programs and decision support systems. We did not find any additional knowledge gain with the additional ''question/ response'' case studies, and we suggest that programs short on funding may consider the costs/benefits of providing this type of feedback which is programming intensive. Our findings do run counter to Casebeer and colleagues who reviewed online continuing medical education modules and found increased knowledge gains in case-based versus text-based interventions. 14 We would also encourage researchers to continue to examine this type of approach-as well as integrating and evaluating other Web-based educational designs. From an authoring perspective, our experiences lead us to recommend that content creators, even those without programming experience, consider using a SCORM-compliant authoring tool. There are many tools on the market for nonprogrammers, such as Powerpoint with the Adobe Breeze Presenter plug-in and Adobe Captivate. Even if the purpose is to publish static content, these tools may increase the accessibility and scalability of the content and support organizational assessment/tracking efforts.
There are a number of design approaches that others may consider when implementing their own online provider education module. One intuitive approach would be to embed links to a toolkit or educational module within the electronic medical record (EMR). Having embedded video with clinician champions explaining course content or with more realistic patient scenarios (simulated laboratory monitors, full motion video, etc) may add utility and provide a more immersive experience. We would encourage others in patient education to consider empirically exploring the design and subsequent effect of education interventions. Ockene and Zapka appropriately noted in their article ''Provider Education to Promote Implementation of Clinical Practice Guidelines'' that increasing the adoption of guidelines need to be multifaceted; provider education alone is not sufficient. Interventions also need to occur at the policy, organizational, and clinical levels. 6 Work is underway at the academic setting where this research took place to integrate reminders for VTE prophylaxis within the EMR and to assess whether patient safety initiatives lead not only to knowledge gains but also to practice patterns. Organizations that utilize computerized physician order entry (CPOE) may consider integrating this type of educational content into the process of order entry.
Our study was limited to residents and fellows. Initially we expected that all medical providers (residents and attending physicians) would be required to complete the online educational module. However, when the time came to launch the intervention it was determined by senior leadership at the medical centers that only residents and fellows would have to take the course in the pilot form. Our findings also only reflect learning at one point in time and not sustained learning over time, nor actual practice patterns and patient outcomes. Questions for both the pre-and posttest were selected by our clinician experts; however, we did not conduct statistical testing to establish validity between the 2. We had a high number (47 of 69) of participants demonstrate mastery at pretest; this may have been an artifact of the pretest being ''easier'' than the posttest. It may have also acted as a sort of ceiling effect; however, a pretest like this could be used by an organization to screen out people who do not need to undertake the full training.
In summary, overall we think it is important that an intervention like this exist-not only to allow organizations to document training for accreditation purposes but also for those providers who want to refresh their knowledge base and as a component of a larger toolkit. Even if it makes one provider more aware of VTE prophylaxis in an organization, that may be one less VTE.