Effectiveness of a Pediatric Emergency Medicine Curriculum in a Public Tanzanian Referral Hospital

Introduction The World Health Organization recently recognized the importance of emergency and trauma care in reducing morbidity and mortality. Training programs are essential to improving emergency care in low-resource settings; however, a paucity of comprehensive curricula focusing specifically on pediatric emergency medicine (PEM) currently exists. The African Federation for Emergency Medicine (AFEM) developed a PEM curriculum that was pilot-tested in a non-randomized, controlled study to evaluate its effectiveness in nurses working in a public Tanzanian referral hospital. Methods Fifteen nurses were recruited to participate in a two-and-a-half-day curriculum of lectures, skill sessions, and simulation scenarios covering nine topics; they were matched with controls. Both groups completed pre- and post-training assessments of their knowledge (multiple-choice test), self-efficacy (Likert surveys), and behavior. Changes in behavior were assessed using a binary checklist of critical actions during observations of live pediatric resuscitations. Results Participant-rated pre-training self-efficacy and knowledge test scores were similar in both control and intervention groups. However, post-training, self-efficacy ratings in the intervention group increased by a median of 11.5 points (interquartile range [IQR]: 6–16) while unchanged in the control group. Knowledge test scores also increased by a median of three points (IQR: 0–4) in the nurses who received the training while the control group’s results did not differ in the two periods. A total of 1192 pediatric resuscitation cases were observed post-training, with the intervention group demonstrating higher rates of performance of three of 27 critical actions. Conclusion This pilot study of the AFEM PEM curriculum for nurses has shown it to be an effective tool in knowledge acquisition and improved self-efficacy of pediatric emergencies. Further evaluation will be needed to assess whether it is currently effective in changing nurse behavior and patient outcomes or whether curricular modifications are needed.


INTRODUCTION
The lack of emergency care systems has been associated with lower survival rates in adults and children in low-and middle-income countries (LMICs). 1,2 At the 72 nd World Health Assembly, delegates recognized the value of emergency and trauma care in reducing morbidity and mortality, and adopted a resolution that would support countries in the development of systems to deliver timely care to critically ill and injured people. 3 In addition to needs assessments and standards for equipment and processes to support the development of emergency care systems, training for all cadres of health workers was one of the identified interventions in the resolution.
Numerous studies surrounding emergency medicine (EM) curriculum implementation have demonstrated decreased mortality in adult populations 4-6 without a significant increase in the use of resources or economic burden. [5][6][7] While nearly 95% of the one million traumatic injuries occurring in children worldwide occur in LMICs, 8 to date, there remains a paucity of open access and comprehensive, pediatric-focused curricula for emergency and trauma care. Furthermore, of the pediatric curricula that have been implemented, most have only evaluated providers' self-efficacy and knowledge acquisition. 9-12 A rare few have attempted to show changes in provider behavior or patient outcomes. 13,14 In response, the African Federation for Emergency Medicine (AFEM) assembled a working group with expertise in pediatric EM (PEM) from seven American and African academic institutions to develop a comprehensive PEM curriculum for three different tiers of healthcare professionals that would be made freely available. The curriculum development process was based upon a widely accepted model for medical education. 15 Curriculum topics were based upon a needs assessment conducted in Tanzania two years prior, and learning objectives were determined by expert consensus review using a modified Delphi process. 16,17 As part of the development process, the implementation and evaluation of the curriculum for the first tier of providers, nurses as described here, was conducted as a pilot study at Muhimbili National Hospital (MHN), the national referral hospital for Tanzania (see Figure 1). Specific efforts were made to broaden the curriculum's evaluation beyond self-efficacy and knowledge acquisition, to include changes in practice behavior.

METHODS
This study was a non-randomized, controlled pilot study to evaluate the effectiveness of a novel PEM curriculum in nurses (Tier 1 providers) by examining the association between participation in this curriculum and nurses' self-efficacy, knowledge, and changes in behavior.

Setting and Study Population
MHN is the national referral hospital for Tanzania and is located in the capital city, Dar es Salaam. It houses an emergency department (ED) that treats approximately 45-50 pediatric patients (under 18 years old) per day.
Nurses were recruited by ED staff at MHN to participate in the training and were matched to control nurses who worked in the same setting based upon their level of experience. All ED Tier 1 nurses or prehospital providers for whom caring for pediatric patients on a daily basis was within their scope of practice were eligible to be enrolled. Any provider not proficient in the English language was excluded from the study. As a retention strategy, a certificate of completion was provided to all participants who attended at least 80% of the training sessions and completed all measurement tools.
The sample-size calculation was based off change in knowledge scores in previously published literature. 7,18 A minimum of 24 nurses (12 intervention and 12 control) was required to detect a 15% change in test scores.

Control Group
As mentioned, the control group of nurses was recruited from the same group of ED nurses as those in the intervention group, and were matched to participants in the intervention group based upon level of experience. The majority of the nurses in both groups (> 60%) hold a diploma in nursing (threeyear program following secondary school or high school), while the remainder possess either a bachelor's degree or a Effectiveness of a Pediatric EM Curriculum in a Public Tanzanian Referral Hospital Chen et al.
certificate in nursing. At baseline, all nurses in the ED receive annual training in pediatric emergencies via the American Heart Association's Pediatric Advanced Life Support course, and a five-day, multidisciplinary course focusing on pediatric resuscitation and trauma management that was developed by local physicians. In addition, they participate in a monthly course entitled "Basic Emergency Nursing Training," which includes some basic pediatric resuscitation training. Further exposure to pediatric-specific training for the nurses is more sporadic, with up to 25% of weekly continuing nursing education sessions being relevant to pediatrics.

Intervention
Training Training sessions were conducted over two and a half days and included 30-minute lectures on nine topics, small-group skills stations and simulated scenarios, as well as scheduled intervals for pre-and post-training measurements and frequent breaks (see Appendix 1). The lectures were delivered by two instructors, one international instructor, and one local instructor, and the small-group sessions were facilitated by an additional three international instructors and two local instructors.

Outcomes/Measurements
The non-randomized, controlled study design was approved by institutional review boards at both the University of California San Francisco (UCSF) and MHN. Outcome measurements focused on three well-described dimensions to evaluate training programs. 19 Primary outcomes were 1) participant self-efficacy (as measured by a survey that used five-point Likert scales to rate participants' confidence with PEM concepts and skills); and 2) participant knowledge acquisition (as measured by written test scores on a 20-question multiple-choice test). Since no surveys for this target audience existed, items were adapted from previously published and validated surveys, with new content development informed by interviews conducted during a needs assessment and review by local experts (EM specialists in low-resource settings). 16,[20][21][22] A similar process for developing questionnaires for educational research has been described. 23 A secondary outcome was change in participant behavior (as measured with a binary checklist of critical actions for pediatric sepsis, respiratory distress, and trauma). Content validity for all tools was obtained through expert review; and a duplicate survey and written test were used for pre-training and post-training evaluations, demonstrating reliability (see Appendices 3 and 4). 24 Inter-observer reliability for the checklist was not assessed. Each group completed all measurements both pre-and post-training, except for the changes in participant behavior, which was only able to be measured post-training (see "Limitations" section). These data were collected for seven weeks post-training.

Statistical Analyses
We used paired t tests to compare means between normally distributed groups and Mann-Whitney U test to compare data that were not normally distributed. Performance on the checklist of critical actions was compared using chi-squared or Fisher's exact tests as appropriate. For all comparisons, a two-tailed p-value less than 0.05 was considered statistically significant.

RESULTS
A total of 15 nurses participated in the training; however, only 11/15 completed all of the pre-and post-training measurements. Fourteen nurses were recruited as control participants; 11/14 completed all measurements. For the survey measurement of self-efficacy, median ratings were similar between the intervention and control groups; however, overall participant post-training ratings were significantly greater than pre-training ratings in the intervention group. Control participants showed no significant difference between their pre-training and post-training ratings (see Tables 1 and 3).
Similarly, for the test of knowledge acquisition, no pretraining difference existed between the intervention and control groups; however, a significant increase in median scores was seen within the intervention group across the two time points, as well as when comparing the intervention group to the control group post-training (see Tables 2 and 3).
A total of 402 live cases of pediatric respiratory distress (121 intervention; 281 control) were observed and measured using the critical actions checklist post-training (Table 4). Only one critical action was observed at a higher proportion in the intervention group (+8.6% (confidence interval [CI], -0.8 -18.1%): "States that the child is in respiratory distress" (which was intended to serve as a proxy for recognition of an emergency condition -see Appendix 2 for the complete tool). For pediatric trauma, 394 live cases were observed (115 intervention; 279 control) with no statistically significant differences in performance of critical actions. For pediatric sepsis, 396 live cases were observed (117 intervention; 279 control). In two related critical behaviors -"States whether the child is or is not anemic" (eg indicating that the nurse checked for anemia) and "Attempts to place IV or IO (if available)" -the intervention group performed these actions at higher rates with estimated differences of + 6.3% (CI, -0.9 -13.7%) and 12.6% (CI, 2.1 -23.0%), respectively.

DISCUSSION
Emergency care has been proven to save lives, and educational curricula have been shown to be one way to effectively and feasibly support the development and expansion of emergency care services in LMICs. [4][5][6][7] Few comprehensive, open-access, pediatric-specific emergency curricula exist despite the high burden of critically ill and injured children in these settings. This study describes the pilot implementation of such a curriculum developed by AFEM to fill this gap, and demonstrates its effectiveness in improving both PEM self-efficacy and knowledge. Once finalized, this curriculum will be made freely available via the Internet to be modified and used to train nurses and prehospital providers across the African continent.
Combined pediatric emergency and critical care fellowships  25 At the same time that sub-specialty training for physicians in LMICs is being conducted at referral centers, it is important to recognize that many emergencies take place far from these large centers.
To ensure that children receive the care they need, nurses and physicians at sites that are further afield also need training in the recognition and initial management of pediatric emergency conditions, but may not have the financial and logistical ability to commit to two or more years of full-time training.
Our curriculum addresses several of these issues. First, while this pilot training included all components and was held over two and a half days, it is designed in a modular fashion so that participants can view the lectures (available both in PowerPoint and PDF formats to accommodate local bandwidth restrictions) at their convenience. The hands-on, small-group skills sessions and simulated cases can be offered in a brief, one-day training with experienced facilitators. This multimodal format has been shown to be preferred by working emergency care nurses, interns, residents, and physicians in a similar setting. 16 Secondly, the final AFEM curriculum is designed to target multiple different cadres of healthcare workers through its tiered development. The Tier 1 curriculum piloted in this study is directed toward nurses and prehospital providers, recognizing that in most LMICs, the majority of the healthcare workforce is made up of professionals who are not physicians.
This study demonstrated a significant improvement in self-efficacy and knowledge scores of participating nurses. As mentioned, multiple studies of educational curricula have shown similar benefits; however, fewer have demonstrated actual changes in behavior. We attempted to show a change in nursing behaviors with this curriculum. However, an improvement in critical actions during specific pediatric emergencies (respiratory distress, trauma, sepsis), which was expected, was not seen for most actions (Table 4).
There are several potential reasons for this. First, critical actions included stating the existence of certain conditions, such as respiratory distress, as a proxy for recognition of the emergency condition. However, observations were being conducted during actual resuscitations and not in a traditional testing environment, so the fact that a nurse did not verbalize critical action statements may not be a true reflection of his or her recognition of these conditions, but rather a reflection of the lack of utility of such statements during live resuscitations. This hypothesis is supported by the observation that rates of performance of critical actions that required statements were relatively low in both groups, ranging from 6-36% (except for "States that the patient is in septic shock"). However, critical actions that followed recognition of these conditions were performed at relatively higher rates, suggesting that nurses may have been acting upon these emergent conditions, even if they were not stating their recognition of them.
Additionally, this pilot study was conducted at only one site (due to a limitation in funding), so there may have been information transfer among the ED nursing staff, which could have led to an improvement in performance of members of the control group, which makes the lack of significance in the rest of the critical actions difficult to interpret. Since pre-training data was not available to help triangulate the results, it is unclear whether participants were unable to transfer the knowledge acquired into action, or knowledge gained spilled over into the control group. The lack of pre-training data also prohibits us from assessing whether the intervention and control groups performed comparably at baseline. However, a significant difference in baseline is unlikely as there was no difference in their pre-training confidence or knowledge scores. The disproportionately higher number of post-training observations in the control group could be due to an unintentional counting of all nurses who did not undergo the training as controls; however, this could not be confirmed due to lack of identifying information.
Other groups have shown that practice change following educational interventions is often difficult to achieve, and our study supports this notion. 5 However, given the knowledge acquisition and improved confidence after the course, we believe that this does not suggest that such curricula are not effective, but, if the data are accurate, might not be sufficient, and that another component of training such as direct oversight  Table 1. Overall median self-efficacy rating (scale 11-55), and within cohort p-value comparison of change in self-efficacy ratings pre-and post-training.  or on-site mentorship is needed. The value of such presence has been stressed by other researchers. 5 Future studies are needed to confirm or refute the lack of translation of knowledge into practice, and if confirmed to examine the effect of direct oversight on practice change.

LIMITATIONS
Our study has several important limitations, including small sample size, possible sample contamination, limited value of specific critical actions as described above, and a failure to confirm inter-observer reliability. In addition, the study design Chen et al.
Effectiveness of a Pediatric EM Curriculum in a Public Tanzanian Referral Hospital was not randomized due to resource limitations and departmental staffing needs; we were provided with a convenience sample of participant nurses that was matched to a group of control nurses. Our pre-training data collection period for observation of critical actions was eliminated due to delays in obtaining IRB approval, making a comparison across the two time periods impossible. In addition, there was selection bias in recruitment, as many of the nurses in the intervention arm had expressed a particular interest in pediatrics and therefore likely had more experience and motivation. The same questions were used pre-and posttraining to assess for knowledge acquisition, which could suggest a positive effect of exposure, however, this is likely limited as the control group did not show a significant increase in scores. Due to funding constraints, this pilot was conducted at only one site, which limits the overall generalizability of the findings.

FUTURE DIRECTIONS
The results of this pilot study have revealed several important areas needing further investigation. As mentioned above, future studies will focus on determining whether true practice change is occurring from this curriculum, as well as the most important factors contributing to behavior change in nurses and physicians in such settings. Interest in re-examining how this curriculum could alter provider behavior and pediatric mortality, especially with larger sample sizes, has already been expressed by institutions in other countries in Africa.
While this study demonstrated an improvement in confidence and knowledge from this curriculum, the ultimate goal is to develop a course that is effective in creating behavior change that leads to a reduction in pediatric mortality. Once we better understand the findings of this research and can make appropriate adjustments to the course, the goal is to make this course freely available to nurses and prehospital providers to download all of the material and adapt it to their needs and their setting. In addition, three local trainers were trained through this pilot study and the hope is that they will later be able to train future trainers to ensure the local sustainability of the course.
The curriculum for Tier 2 providers (eg, clinical officers, intern physicians) is currently being created and will be piloted at a large medical center in West Africa. The results of this study and subsequent planned studies will be used to modify this curriculum as well. Once these two tiers have been modified and piloted, the curriculum for Tier 3 providers (eg, specialist physicians) will be created and similarly evaluated.