High-Fidelity Simulation Enhances Pediatric Residents’ Retention, Knowledge, Procedural Proficiency, Group Resuscitation Performance, and Experience in Pediatric Resuscitation
Objective: The goal of this study was to assess the effect of high-fidelity simulation (HFS) pediatric resuscitation training on resident performance and self-reported experience compared with historical controls.
Methods: In this case-control study, pediatric residents at a tertiary academic children’s hospital participated in a 16-hour HFS resuscitation curriculum. Primary outcome measures included cognitive knowledge, procedural proficiency, retention, and self-reported comfort and procedural experience. The intervention group was compared with matched-pair historical controls.
Results: Forty-one residents participated in HFS training with 32 matched controls. The HFS group displayed significant initial and overall improvement in knowledge (P < .01), procedural proficiency (P < .05), and group resuscitation performance (P < .01). Significant skill decay occurred in all performance measures (P < .01) with the exception of endotracheal intubation. Compared with controls, the HFS group reported not only greater comfort with most procedures but also performed more than twice the number of successful real-life pediatric intubations (median: 6 vs 3; P = .03).
Conclusions: Despite significant skill decay, HFS pediatric resuscitation training improved pediatric resident cognitive knowledge, procedural proficiency, and comfort. Residents who completed the course were not only more proficient than historical controls but also reported increased real-life resuscitation experiences and related procedures.
The Accreditation Council for Graduate Medical Education requires pediatric training programs to certify competency in critical care procedures and emergency situations.1 The Review Committee for Pediatrics defines specific procedural competencies that programs must document for each resident physician.2 Historically, residents have gained resuscitation experience in the emergency department, ICUs, inpatient wards, and via the Pediatric Advanced Life Support (PALS) course.3–5 However, most residents finish training with inadequate knowledge and proficiency with critical care skills.5–11 Surveys of pediatric residents reveal that large percentages have never led a resuscitation event, and program directors perceive that many fail to achieve procedural competence.6,11,12 The PALS course is insufficient in ensuring a resident’s prolonged mastery of resuscitation skills.13–17 With duty hour restrictions limiting exposure to critically ill patients, incorporating adjunctive educational methods is essential.
Simulation has the potential to measure and improve residents’ resuscitation skills, procedural proficiency, comfort, teamwork, and communication.18–33 High-fidelity simulation (HFS) uses computerized, interactive, life-sized pediatric manikins that offer realistic patient responses. HFS is gaining acceptance in medical training because of its safety, reproducibility, and ease of simulating critical events, and some studies have suggested that simulation is superior to traditional experiential training.34–38 As such, the standard practice of “see one, do one, teach one” may be evolving into “see one, simulate many, do one competently, and teach everyone.”37
Few studies have measured pediatric residents’ retention after HFS training. The objectives of the current study were to: (1) evaluate the effectiveness of HFS training on cognitive knowledge and observed performance; (2) assess long-term retention by retesting the following academic year; (3) assess the effect of retention interval on skill decay of cognitive knowledge and observed performance; and (4) compare cognitive knowledge, observed performance, comfort, and self-reported real-life experience between residents who complete HFS training versus historical residents with no previous simulation training.
Study Design/Study Population
This study was conducted between August 2008 and February 2011. Participants were pediatrics and medicine-pediatrics residents during their first (postgraduate year [PGY]-1) through third (PGY-3) year at the Medical University of South Carolina, a 173-bed quaternary care hospital with ∼22 000 pediatric emergency department visits, 700 PICU patients, and 1000 NICU patients per year. Residents complete 5 months of training in the pediatric emergency department, 3 months in the NICU, and 1 month in the PICU with 2 months of PICU cross-cover. Residents participate in the Pediatric Medical Emergency Team and Mayday response teams when on-call on the wards and PICU rotations. All participants completed PALS and Neonatal Resuscitation Program training before starting their residency and before study participation.
This study was approved as exempt research by the institutional review board of the Medical University of South Carolina, and informed consent was therefore not required.
The HFS training program used a 16-hour pediatric resuscitation course (Appendix 1), split into two 8-hour days separated by 2 weeks. The curriculum was developed at the Emergency Medicine Foundation and the American College of Emergency Physicians Teaching Fellowship Program (August 2000–March 2001; Dallas, Texas) and was reviewed by fellowship faculty to ensure that it met the standards of the Instructional System Design model for educational instructional development.39–42 The course was piloted in 2001 and 2002, and the results were published previously.32 The pilot course was revised for use with HFS in 2008. The learning objectives, evaluation procedures, and instructional strategy were based on 2005 American Heart Association pediatric resuscitation guidelines.43
Eight to 10 courses were given each academic year to ensure that all residents were trained. Groups consisted of 6 residents with 2 instructors. The curriculum director performed instruction and assessment with assistance from the pediatric chief residents and a pediatric nurse practitioner. Courses were videotaped and reviewed by the curriculum director to ensure interobserver consistency. The HFS curriculum consisted of online didactics, surveys, cognitive knowledge assessment, hands-on manikin training, and procedural knowledge assessment (SimBaby; Laerdal Medical, Stavanger, Norway). On day 1, participants were trained in procedural proficiency and group resuscitation by using deliberate repetitive practice until mastery was achieved. Mastery was defined as error-free performance. On day 2, additional practice was provided to achieve a high degree of overlearning. Overlearning was defined as deliberate overtraining of skills past the set criterion performance level of mastery.44
Data Collection/Outcome Measures
Participants were enrolled as PGY-1 or PGY-2. Appendix 2 lists the performance measures analyzed. Cognitive knowledge of pediatric resuscitation was assessed by using a 50-question multiple choice examination. Observed performance (procedural proficiency and group resuscitation performance) was assessed through direct observation by using structured checklists, which consisted of discrete observable critical actions that were dichotomously scored as “performed correctly” or “not performed correctly.” Summative performance was scored as the percentage of critical actions correctly performed. Group performance was assessed during a standardized patient scenario of pulseless arrest and evaluated teamwork and patient management skills. The selection of procedural proficiency skills (Appendix 2, Part II) to be trained and tested was based on the 2006 Pediatric Residency Review Committee’s “Table of Technical and Therapeutic Procedures.”45 Skills that were listed as training requirements (the resident should master the performance of or be developing skill with the procedure) were emphasized. The content of the cognitive knowledge test and the structured observed checklists was based on 2005 American Heart Association guidelines for PALS.43 Self-reported comfort and real-life experience with resuscitation skills were assessed by using a confidential survey. A 7-digit Likert scale (from “significantly uncomfortable” to “significantly comfortable,” with a “neutral” midscale) was used for self-reported comfort levels.
Performance measures were assessed before initial training (time 1) and immediately posttraining (time 2). Retesting (time 3) occurred the following academic year before retraining. Self-reported experience was assessed at times 1 and 3. Primary outcome measures included initial improvement (time 1 to time 2), skill decay (time 2 to time 3), and overall improvement (time 1 to time 3). Retention interval was defined as the time interval between times 2 and 3. Retention was defined as the absolute performance at time 3 compared with time 2. Skill decay was defined as the decrease in performance from time 2 to time 3. The effect of retention interval on skill decay was assessed by comparing study participants who retested at ≤12 months after initial training versus those who retested at >12 months. Correlation between self-reported comfort and real-life experience was analyzed in the intervention group.
Historical Control Group
To account for the confounding factor of residency training on knowledge and skill retention, residents who participated in the HFS curriculum (ie, the intervention group) were compared with historical residents who did not participate (ie, the control group) by using a matched-paired cohort study. Matching was based on years of training. The control group was tested with the same performance measures (Appendix 2) in the same HFS environment as the intervention group. A subgroup analysis was performed in which PGY-2 control residents were compared with PGY-2 intervention residents and PGY-3 controls with PGY-3 intervention residents. Comparisons of cognitive knowledge, observed performance, comfort levels, and self-reported real-life experiences were assessed between groups. This comparison occurred at time 3 for the intervention group to determine the effect of the HFS course on retention.
The study was implemented to augment the procedural and resuscitation training curriculum in a pediatric residency program. As such, all residents were required to participate in the study. A retrospective power analysis was performed. The study was powered to detect at least a 3.25-point difference in procedural assessment scores at time 1 and time 2 with an α of .05 and 95% power. Assumptions used were normal distribution of scores and a historical SD for the scores of 5.3 points.46
Descriptive statistics were calculated for cognitive knowledge, procedural proficiency, and group resuscitation performance at times 1, 2, and 3 for the intervention group. Similar control group statistics were collected. Correlation coefficients were generated to assess the relationship between self-reported comfort and experience levels in the intervention group.
Wilcoxon signed rank tests and paired t tests were used to test differences between cognitive knowledge, procedural proficiency, or group resuscitation performance of the intervention group when appropriate. Repeated analysis of variance was completed to assess within-subject effects of the intervention group. All statistical calculations were performed by using SAS version 9.3 (SAS Institute, Inc, Cary, NC).
Forty-one pediatrics and medicine-pediatrics residents were enrolled in the intervention group. The mean length of completed residency training at the time of initial HFS training and retesting was 11.2 months and 23.5 months, respectively. The median retention interval (time 2 to time 3) was 12 months (range: 2–24 months). Thirty-two matched-pair historical control residents (mean length of training: 23.9 months) were selected for comparison at time 3. Table 1 compares demographic characteristics of the intervention and control groups.
Effect of Initial Training, Skill Decay, and Overall Improvement
Tables 2, 3, and 4 show the effectiveness of HFS training on initial improvement, overall improvement, retention, and skill decay. Baseline proficiency in all skills (time 1) was poor despite previous PALS training. The median time between PALS training and initial HFS training was 10 months (mean: 11.2 months). There was significant initial and overall improvement in all performance measures. Retesting the following academic year, however, showed significant skill decay of all performance measures with the exception of endotracheal intubation. Skill decay of procedural proficiency was significantly less than cognitive knowledge (P < .01) and group resuscitation performance (P < .01). There was no difference in performance between residents who retested at >12 months versus those retested ≤12 months after the initial course. There was no difference in mean months of training between residents who retested at >12 or ≤12 months (23.7 vs 23.4 months, respectively). Repeated analysis of variance testing demonstrated that cognitive knowledge and procedural proficiency scores showed significant effects, with scores increasing over time (P < .01).
Comparison of Intervention and Historical Control Groups
Residents who had HFS training the previous academic year demonstrated significantly greater proficiency in cognitive knowledge and all observed performance measures, as seen in Tables 5, 6, and 7. Subgroup analyses comparing PGY-2 residents in both intervention and historical control groups and PGY-3 residents in both groups revealed the same significant differences.
Compared with controls, the intervention group reported greater comfort and more actual experience with pediatric resuscitations and procedures. Comfort levels were significantly higher for all skills except resuscitation of shock requiring fluid resuscitation, evaluation and management of dysrhythmias, and bag-valve-mask ventilation with and without chest compressions (Table 8). The intervention group reported more successful real-life intubations than controls (median: 6 vs 3; P = .03) and more experience with resuscitation of pulseless arrest (median: 1 vs 0; P = .02). Experience with respiratory failure approached but did not attain significance (median: 4 vs 3; P = .06).
Correlation Between Resident Comfort and Experience
Significant moderately positive correlations (r = 0.3–0.7 with P < .05) existed between comfort levels and self-reported real-life experience with most pediatric resuscitation skills in the intervention group. The only exceptions were with chest compressions (r = 0.18), intraosseous needle insertion (r = 0.17), and termination of resuscitation (r = 0.26). Endotracheal intubation success displayed a strong correlation (r = 0.72) between comfort and self-reported experience.
This study contributes to the field of pediatric simulation research in several aspects. Primarily, the results add to the existing body of research that simulation training has positive effects on resuscitation education.
It demonstrated that an HFS pediatric resuscitation curriculum not only significantly improved residents’ cognitive knowledge and procedural proficiency but also increased their self-reported frequency of performing resuscitation skills in actual clinical practice. To achieve these results, the study design that was used differed in approach compared with many previous studies. The inclusion of the historical control group addressed the possible confounder of traditional experiential training on retention and the possibility of the Hawthorne effect (the notion that the observed performance of participants may be influenced by the fact that they knew they were being observed). Using the historical control group was advantageous compared with a pre-/post- study design in that both the intervention and control groups were subjected to the Hawthorne effect. Thus, the fact that intervention residents demonstrated greater proficiency in cognitive knowledge and observed performance, higher levels of comfort, and more actual experience with pediatric resuscitations and procedures than the control group could be attributed to the intervention alone. Although a randomized controlled study would be superior to the use of historical controls, using a control group with traditional training alone was considered to be unethical given the positive preliminary results of simulation on both resuscitation performance13,18–21,47–52 and procedural proficiency.22–27,33
Before this study, research investigating the long-term effects of simulation training on pediatric residents’ retention of knowledge and procedural skills yielded variable results.18,20,21,23,24,27 The current study demonstrated that skill decay occurs despite the novel modality of HFS but found that retention varied based on the skills that were assessed. Group resuscitation performance deteriorated to a greater degree than cognitive knowledge, and both declined more than procedural proficiency. Some skills decayed more significantly than others (ie, cardioversion/defibrillation versus endotracheal intubation) and could warrant more frequent retraining. These observations may represent the underlying complexity of group dynamics as well as highlight the effect of frequency of exposure in clinical practice. The degree of skill decay was unchanged when comparing retention intervals of ≤12 months versus >12 months, which implies that the decrease in performance occurred soon after training but then remained relatively static for the range of retention intervals (2–24 months). Because this study was not designed to determine the optimal retraining interval to minimize clinically significant skill decay, further research is needed in this area.
There were several limitations to the current study. Baseline performance scores for the control group were not collected because the study was started during their second year of residency. Baseline assessments of control and intervention groups at the start of residency training would have controlled for differences due to inherent cognitive and procedural competence. Although cognitive knowledge testing, self-reported comfort levels, and real-life experience were collected in a blinded and confidential manner, the observed performance measures (procedural proficiency and group resuscitation performance) were unblinded. The lack of blinded assessors may have favored finding a difference, despite the use of structured checklists. The intervention group’s familiarity with the testing content from the previous year may have accounted for their improved performance compared with controls. However, the independent measures of self-reported real-life experience and comfort levels support the inference that residents are more comfortable attempting procedures during actual resuscitation events after completing simulation training, a necessary prerequisite for proficiency.
Another limitation is that this study was conducted at a single academic institution with a relatively small number of participants. A multicenter trial with a larger cohort would provide greater validity. Scenarios also included only resident trainees; including multidisciplinary caregivers would allow a more realistic environment in which to assess team performance.
This study did not include the use of validated measurement tools to assess resident performance, which remains a universal limitation of simulation research. However, increasing evidence suggests that valid and reliable assessments of procedural skills may be achieved by using simulation, and the data support the use of structured checklists.6,53–57 The individual and group procedural skill checklists used in this study were based on the 2005 American Heart Association guidelines for PALS, giving them a degree of content validity. However, formal validation of the observed structured checklists was not performed, thus limiting the construct validity of our performance measures. To minimize interobserver variability, the curriculum director oriented all instructors and directly observed or reviewed the participants’ videotaped recordings. The lack of validated measurement instruments that are discriminatory, reproducible, and reliable (interobserver and intraobserver) is a major focus of the multidisciplinary simulation research networks. The Examining Pediatric Resuscitation Education Using Simulation and Scripting and the Patient Outcomes in Simulation Education collaborative groups are currently assessing the development and validation of scoring tools, which may facilitate a standardized approach across institutions.58,59
Pediatric residents who participated in a HFS resuscitation course demonstrated long-term improvement and retention in cognitive knowledge, procedural proficiency, and group resuscitation performance compared with the historically matched cohort with only standard residency training. Residents who participated in HFS training reported an increase in comfort and real-life experience with resuscitations and procedures compared with historical controls. Because significant skill decay occurs over time, retraining is needed to maintain high proficiency standards.
APPENDIX 1 Pediatric Resident Simulation Training Curriculum
APPENDIX 2 Performance Measures
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
- high-fidelity simulation
- Pediatric Advanced Life Support
- postgraduate year
- 1.↵ACGME Program Requirements for Graduate Medical Education in Pediatrics. Available at: www.acgme.org/acgmeweb/ProgramandInstitutionalGuidelines/MedicalAccreditation/Pediatrics.aspx. Accessed April 30, 2013.
- 2.↵The American Board of Pediatrics. A guide to evaluating your clinical competence in pediatrics. Available at: https://www.abp.org/abpwebsite/publicat/respamphlet.pdf. Accessed November 6, 2012.
- Mulvey HJ,
- Ogle-Jewett EA,
- Cheng TL,
- Johnson RL
- Halamek LP,
- Kaegi DM
- Buss PW,
- McCabe M,
- Evans RJ,
- Davies A,
- Jenkins H
- Gaies MG,
- Landrigan CP,
- Hafler JP,
- Sandora TJ
- Hunt EA,
- Hohenhaus SM,
- Luo X,
- Frush KS
- Berden HJ,
- Willems FF,
- Hendrick JM,
- Pijls NH,
- Knape JT
- Gaies MG,
- Morris SA,
- Hafler JP,
- et al
- van Schaik SM,
- Von Kohorn I,
- O’Sullivan P
- Niebauer JM,
- White ML,
- Zinkan JL,
- Youngblood AQ,
- Tofil NM
- Dobson J,
- Brancati D,
- Nagel R
- McCombs B
- Dick W
- Banathy B
- 42.↵Gagne RMBL. Principles of Instructional Design. 2nd ed. New York, NY: Holt, Rinehart, and Winston; 1979.
- 45.↵Table of Technical and Therapeutic Procedures. Available at: www.ambpeds.org/egwebnew/procedures/tablettproc.cfm?trackerid=1027. Accessed November 6, 2012.
- Boet S,
- Borges BC,
- Naik VN,
- et al
- Hunt EA,
- Walker AR,
- Shaffner DH,
- Miller MR,
- Pronovost PJ
- Falck AJ,
- Escobedo MB,
- Baillargeon JG,
- Villard LG,
- Gunkel JH
- Cheng A,
- Hunt EA,
- Donoghue A,
- et al
- Donoghue A,
- Ventre K,
- Boulet J,
- et al
- Copyright © 2013 by the American Academy of Pediatrics