Assessing the Effect of an Intensive 2-Week Surgical Training and Innovation Program for High-School Students

OBJECTIVE: The summer surgery program (SSP) was founded in 2012 as an educational program for students at the critical juncture between high school and college to engender interest in medicine, science, and innovation. This program has a distinct emphasis on innovation and problem solving based on real-life operative challenges identi ﬁ ed by students during surgical observation in the operating room. The effect of the SSP regarding postsecondary education and career goals was evaluated by participants using a follow-up questionnaire. DESIGN: Retrospective cohort study using web-based survey administered to students at least 1 year after participation in the SSP. Associations between demographics and survey responses were made using Fisher ’ s exact test and a Bonferonni correction was used to account for multiple comparisons. PARTICIPANTS: Between July 2012 and August 2015, 119 students enrolled in the SSP. We sent a web-based questionnaire link to all participants who completed the program. The questionnaire contained 80 questions assessing the participant ’ s interest in studying medicine or science in college, knowledge of health care, and their appreciation and understanding of innovation. SETTING: UC Irvine Medical Center, Orange, CA; Institutional tertiary care center. RESULTS: In total, 77 (64.7%) of 119 students who matriculated in the SSP completed the follow-up survey; the mean number of years after the program was 2.09 years. Nearly all students reported the program increased their interest in studying medicine or science in college (97.4%), led them to a better understanding of their own career goals (93.5%) and made them more con ﬁ dent in their ability to succeed in a career in health care (88.3%). The majority indicated the program led them to better understand the training and schooling required of doctors and surgeons (94.8%), and led them to better appreciate the roles of different medical specialties (96.1%). Overall 96% of students reported that the program led them to better understand the importance of innovation and 86% of the respondents noted they better understood the process of innovation. Participants in the SSP were con ﬁ dent they would be able to become a health professional (p o 0.0001). Of note, there was no drop off in the ratings for the program when comparing classes that were 1, 2, 3, or 4 years after their SSP experience. CONCLUSIONS: The follow-up survey revealed that the 2 week SSP had a markedly, long lasting positive effect on participants in areas of academic, career, and innovation-related variables. ( J Surg Ed ] : ]]] - ]]] . J C 2017 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

RESULTS: In total, 77 (64.7%) of 119 students who matriculated in the SSP completed the follow-up survey; the mean number of years after the program was 2.09 years. Nearly all students reported the program increased their interest in studying medicine or science in college (97.4%), led them to a better understanding of their own career goals (93.5%) and made them more confident in their ability to succeed in a career in health care (88.3%). The majority indicated the program led them to better understand the training and schooling required of doctors and surgeons (94.8%), and led them to better appreciate the roles of different medical specialties (96.1%). Overall 96% of students reported that the program led them to better understand the importance of innovation and 86% of the respondents noted they better understood the process of innovation. Participants in the SSP were confident they would be able to become a health professional (p o 0.0001). Of note, there was no drop off in the ratings for the program when comparing classes that were 1, 2, 3, or 4 years after their SSP experience.

INTRODUCTION
The interest and attitude of high-school students in the United States (U.S.) toward science and achievement appear to decline before their entry into college. 1 In addition, these same students have a lack of high-school science preparation compared with students in many other first world countries. The 2009 National Assessment of Educational Progress reported that only 21% of high-school seniors scored at or above proficiency in science education. 2 To address this concerning trend, there has been an increase in the number of health-related, nonschool-based biomedical pipeline programs for high-school students. 3 Summer biomedical pipeline programs incorporate an inquiry-based, interactive learning environment and have been shown to effectively promote critical reasoning skills and increase a student's interest in pursuing science-related careers. 4,5 Universityaffiliated programs hosted in large medical centers offer participants access to diverse clinical settings, mentoring by faculty and student role models, as well as access to research and clinical skills centers. This kind of exposure may increase a student's awareness of science-and healthrelated careers and thereby alter their subsequent professional path. Indeed, the success of programs of this nature has been manifested by high percentage of participants attending college, majoring in biological and physical sciences and pursuing medical or graduate school. 6 When students search for summer medical experiences, they will largely choose between 2 well-defined prototypical biomedical pipeline program formats. In summer research programs, students are exposed to research and innovation through direct involvement in a biomedical research setting. Students assist in laboratory-based research and are often asked to share their research contributions in an end-of-program symposium. In contrast, summer medical programs facilitate health care experience by hosting lectures, teaching basic medical procedures, and offering operating room (OR) shadowing. The clinical focus of these programs allows for direct patient interaction and exposure to diverse health care settings.
The University of California, Irvine (UCI) Summer Surgery Program (SSP), founded in 2012 by the Department of Urology, is distinguished by its emphasis on both surgical exposure and innovation. The surgical program incorporates an intense in-depth curriculum focused on surgical anatomy, technical skills, an introduction to innovative techniques (i.e., laparoscopy, robotic surgery, and endoscopy) and mentorship. Participants shadow surgeons in the OR and receive hands-on surgical skill instruction in UCI's Surgical Skills Center. The program's innovation focus, centered within the innovator group project, requires that students, after identifying surgical challenges in the operating theater, develop innovative solutions, which in turn are presented to biotechnology experts at an end-ofprogram symposium. Projects are evaluated by experts based on feasibility, design, and marketability.
To our knowledge, the only other surgery-focused summer program in the USA is the Stanford cardiothoracic surgical skills program that primarily focuses on teaching surgical skills. Notably, the Stanford program and other summer medically focused programs do not incorporate innovation into their primary focus. 7,8 The UCI-SSP was evaluated by its participants in a follow-up questionnaire targeted to those students completing the program during its initial 3 years. This study aims to assess the type of effect the program had on its participants as it relates to postsecondary education, career goals, and exposure to innovation in medicine.

Program Background and Eligibility
The SSP was founded by the UCI Department of Urology in 2012 with the aim of providing students a comprehensive introduction to surgery with an emphasis on the importance surgical innovation. Two 2-week sessions with total enrollment of 40 to 50 students are hosted each summer. Competitive enrollment was based on the following: current matriculation in high school, age Z16 years, a minimum 3.5 grade point average requirement, strength of 2 letters of recommendation, quality of responses to the application questions and a phone interview. On average, 100 applications are received and 48 students are accepted yearly. Four students each year were granted scholarships and the remainder paid tuition to attend the program as it is the normal practice for summer medical experience programs.

Program Curriculum
Program days are comprised of a morning and an afternoon session (Appendix A). The morning sessions begin with "Pre-Rounds" lectures given by UCI surgical faculty on various surgical, clinical, and research-oriented topics. This is followed by an anatomy course comprised a 5-session fetal pig dissection with concordant human anatomy lectures. Two innovator groups combine and alternate between two 2-hour sessions in the Department of Urology Surgical Skills Center and the UC Irvine Douglas Hospital OR. In addition, students become certified in basic life skills training, complete an ultrasound curriculum and participate in various didactic sessions on innovation, radiology, and case discussions.
In the skills laboratory, students receive instruction from faculty mentors on surgical skills such as open knot tying and suturing along with instruction in laparoscopic, endoscopic, and robotic (Da Vinci, SI, Intuitive Surgical, Sunnyvale, CA) techniques. In the OR the students observe surgical procedures performed by faculty from multiple departments including general surgery, orthopedics, urology, and neurosurgery. OR shadowing varied from day-today and did not necessarily correspond with the daily didactics. The students have the opportunity to interact with the surgeons to discuss real-life applications of surgical skills and begin to develop an understanding of the role and limitations of contemporary technology in current surgical practice. The program concludes with a final examination including an anatomy practical, laparoscopic skills assessment, and written examination. The innovator group project is an equally important component of the program. Participants are assigned to 1 of 4 innovator groups at the outset of the program. The groups are tasked to develop an innovation to resolve an inefficiency or surgical challenge noted during live surgical observation or during surgical skill instruction sessions. Innovator group breakout sessions occur daily to allow participants an iterative innovative process. Student group innovations are presented in a symposium on the final day of the course.
Multiple layers of mentorship are provided in the program to optimize the participating high-school students' experience. UC Irvine medical students and undergraduate premedical volunteers provide junior mentorship in the innovator groups. These leaders underwent an orientation outlining their expectations as instructors and were encouraged not to provide their own ideas but rather to cultivate the innovative ideas proposed by the students. The comprehensive curriculum is led by medical student leaders who participate in didactic teaching and anatomy dissection. Daily breakout sessions revolve around reflection and predetermined discussion points. In addition, a session on college admissions occurs during the 2-week course. Senior mentors include surgical faculty who have experience with surgical device and surgical technique development, clinical trials and interaction with biotechnology companies. The senior mentors oversee the program and interact with students on a daily basis to encourage their creative potential.
UC Irvine medical students designed the curriculum in conjunction with senior UC Irvine Urology faculty members. The curriculum was designed to limit passive learning and maximize exposure to surgical shadowing, hands-on surgical skills training, and innovation. Participant feedback resulted in small adjustments to the curriculum from year-to-year.

Follow-Up Survey
UC Irvine Institutional Review Board approval was obtained and a web-based survey of 80 questions was administered to student alumni who were 18 years or older having participated in the program over the past 4 years (i.e., 2012-2015). The aim of the survey is to understand the effect of the program had on the students' understanding of medicine, their career choices, the effect of their academic and clinical involvement, and their confidence/ comfort level in clinical settings as well as their appreciation and understanding of innovation. A total of 119 students received the survey (Supplemental attachment 1).

Statistical Analysis
Student characteristics, including demographics and education, and parental characteristics were analyzed ( Table 1). The proportion of participants who were positively influenced by the SSP along with 95% confidence intervals was reported by the primary outcomes regarding diverse aspects of influence (Table 2). The primary outcomes questionnaire included responses to questions regarding knowledge of medicine, interest in studying medicine or science in college, knowledge of health care, understanding of individual career goals, confidence to succeed in a career in health care, and confidence to participate in health-related extracurricular activities. Using Fisher's exact test, we assessed the association between the highest degree students imagined they would achieve and their parents' demographic characteristics (marital status, highest education level, and having a physician parent). For the secondary aims, Fisher's exact test was used to assess how the participants' experience in research and surgery-related job/ volunteering was associated with whether the participant desired to become a physician or otherwise pursue a career in a health-related field. Similarly, we tested if a positive experience in the SSP was associated with each participant's self-evaluation; the responses to these questions were as follows: strongly disagree, disagree, neutral, agree, and strongly agree. Bonferonni correction was used to account for multiple comparisons. The analysis was conducted in SAS version 9.3 (Cary, NC). Our data analysis was designed to determine the following: (1) the proportion of participants who were positively influenced by the SSP; (2) the association between the professional degree to which participants aspired and their parents' demographic characteristics; and (3) the association between the professional degree they would seek to complete and primary survey outcomes. Our secondary aims included the following: (1) the association between participants' experience and their desire to become a doctor or to pursue a career in a health-related field and (2) the association between participants' self-evaluation and primary effect outcomes.

Descriptive Summary of Participants' Characteristics
SSP participants came from Thailand, Hong Kong, the Netherlands, Mexico, and Taiwan as well as throughout the United States (Massachusetts, Illinois, Minnesota, Virginia, and California).
In total, 77 of 119 students previously enrolled in the UCI-SSP completed the follow-up survey (65% response rate). Further, 8 surveys out of 15 (53%) were completed by students whom attended in 2012, 19 out of 35 (54%) by 2013 students, 30 out of 46 (65%) by 2014 students, and 20 out of 23 (87%) by 2015 students. Over half of the SSP participants were female (65%). At the time of follow-up, most participants had grade point average greater than 3.5 (77.9%), taken more than 4 AP/IB courses (75.3%) and, of those eligible, all were attending college (100%). The proportion of participants who planned to pursue a graduate degree (PhD, MD, etc.) was 87.0%. Furthermore, 53.0% of participants' fathers had a level of education beyond a college degree (e.g., Master's//Doctorate/MBA/MD/DDS) and similarly 44.2% of participants' mothers had a graduate degree. Only 16% of participants' parents were medical doctors. Also, 20% of parents had no college degree, and 16% of students came from a family in which neither parent had attended college.

Assessment of Program Effect
Nearly all students were positively influenced by the program, reporting that the program increased their knowledge of medicine (98.7%), increased their interest in studying medicine or science in college (97.4%), and helped them understand health care better (97.4%) ( Table 2). Also, nearly all indicated that the program led them to a better understanding of their own career goals (93.5%), made them more confident in their ability to succeed in a career in health care (88.3%), increased their confidence in their ability to participate in health-related extracurricular activities (such as volunteering or shadowing) (89.6%), influenced their decision to pursue medical school (87.0%), led them to better understand the training and schooling required of doctors and surgeons (94.8%), led them to better appreciate the roles of different medical specialties (96.1%), and led them to better understand the lifestyle of a doctor and surgeon (92.2%). Three-fourths of the students indicated that they had selected a science-related major in college.
Most participants reported a positive experience regarding innovation (Table 3). Indeed, 96% students reported that the program led them to better understand the importance of innovation and 86% reported that the program led them to better understand how to innovate. One project originally presented at the innovator group project symposium became the subject of further study and led to publication in a peer-review journal. 9 The educational degree that participants planned to seek was not significantly associated with parental demographic characteristics (Table 4). Participants with at least one parent with a graduate degree were more likely to agree or strongly agree to the statement that the program led them to a better understanding of their career goals (p ¼ 0.0464) ( Table 5). Prior experiences in research were not associated with whether they wanted to become a physician (p ¼ 0.4975) or whether they wanted to pursue a career in a healthrelated field (p ¼ 0.8514). The participants' experiences in surgery-related job/volunteering were also not associated with whether they wanted to become a physician (p ¼ 0.3384) or to pursue a career in a health-related field (p ¼ 0.7470) ( Table 6). The group of participants who believed that the SSP made them more confident in their ability to succeed in a career in health care had significantly more confidence in becoming a health professional (p o 0.05) ( Table 7). There was no significant difference between A, neither parent is college educated; B, at least one parent is college educated and highest education is college; C, at least one parent is with a graduate degree. *p Value here is based on Fisher's exact test (due to small size in some cells). *Indicates statistical significance, p o 0.05. responses from students graduating from the program at different years (Table 8).

DISCUSSION
The UCI-SSP had a positive effect on most of the attendees, even several years after the exposure. At baseline, the student participants of the SSP were already very academically motivated, with 87% citing plans to obtain a graduate degree. The participants came from educated families with 53% and 44% having fathers and mothers who had obtained a Master's, Professional, or Doctorate degree; indeed only 16% of participants came from a family where neither parent was college educated and 16% came from a family where at least one parent was a medical doctor. Of note, more than half of the participants were female; this is consistent with the rise in female matriculation into U.S medical schools over the past several decades. 10 At the time of the survey, 100% of students eligible for college were attending college with 75% pursuing a major in a biology-related field. Although it is reasonable to expect that the highly qualified students who were selected into the SSP would proceed to college, we believe it is notable that 41% of these talented students noted that the SSP provided direction that altered their college class selection and 93% agreed their SSP experience resulted in a better understanding of their individual career goals. Participants coming from a household with one or both parents having a postgraduate degree were more likely than those from households with neither parent attending college to agree that the program led them to a better understanding of their career goals. This either suggests that participants whose parents had less education already had clearly defined career goals or that participants whose parents are more educated were more able to refine their career options after the SSP experience. Differences in participants' career aspirations were not associated with more or less benefit from the program. Firsthand observation in the multidisciplinary setting of the OR and direct interaction with surgeons of diverse specialties resulted in 490% of students reporting that the program provided them with a better understanding of the training, schooling, and lifestyle of doctors and surgeons as well as a better appreciation of the roles of different surgical specialties. In addition, participants who acknowledged a positive effect of the program on their confidence level also demonstrated greater confidence in becoming a health professional than those students who felt their confidence was not influenced by the program. We believe that several elements of the SSP were critical to the students' comfort with the health care setting including the direct OR observation time, the handson surgical skill exposure and the exposure to medical professional mentorship at several different levels.
The SSP corroborated the value of incorporating innovation into a hands-on surgical experience. To the best of our knowledge, the only other exclusive surgery-focused skillsbased summer program is the Stanford cardiothoracic surgical skills. In contrast to the Stanford program, the UCI-SSP curriculum linked the students' clinical experience with many of the practical steps related to innovation, particularly regarding identifying a problems and critically proceeding to devise a solution in a team-based atmosphere. In the follow-up survey, 96% and 86% of students noted that the program led them to better understand the importance of innovation and how to innovate, respectively. Of note, one of the student's projects led to a formal study and subsequent publication of an article in a peer-reviewed journal. Importantly, the SSP had a sustained effect that did not wane over the course of 4 years.
There are several limitations to our survey. First, the results could represent a selection bias as only those that had a favorable experience in the SSP may have responded to the survey. The assessment of the percentage of students going to college was limited by our ability to determine those that were at the appropriate age to attend college. In future years, the National Student Clearinghouse will be used to track educational enrollment and outcomes. In addition, the most effective method of assessing the program's effect would most certainly by the administration of a preprogram survey analysis, which could then be compared with the postprogram survey. Starting in 2015 the SSP has implemented a pre-SSP survey for all incoming students.

CONCLUSION
The UCI-SSP had a positive effect on participants in areas of academic and career variables. Exposure to innovation and its principles led to a sustained positive outlook with respect to each participant's assessment of their individual innovative potential.

ACKNOWLEDGMENTS
We greatly appreciate the assistance of Dr Martin Hoffman for his work in development of the SSP curriculum, Ms Katherine Bergin for her organization of the program, Renai Yoon for his tireless efforts in organizing lab activities, as well as the SSP volunteers and OR staff. We also would like to thank Dr Ralph V. Clayman for his assistance in the development of this manuscript and providing us all with the inspiration to pursue excellence in our academic endeavors. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

APPENDIX A
Schedule of OR Shadowing and Surgical Skills Training, Days 2, 3, 5-7, and 9. Closing ceremony breakfast with family 9:00 AM-10:15 AM Innovator group presentations (4Â) -10 min/group 10:15 AM-11:00 AM Innovator group and student awards* 11:00 AM-12:00 PM Final remarks, graduation, and photos *Awards given to student with highest score on anatomy practical and surgical skills examination.