Implementation of a Modified WHO Pediatric Procedural Sedation Safety Checklist and Its Impact on Risk Reduction
BACKGROUND AND OBJECTIVES: Major adverse events (AEs) related to pediatric deep sedation occur at a low frequency but can be of high acuity. The high volume of deep sedations performed by 3 departments at our institution provided an opportunity to reduce variability and increase safety through implementation of a procedural sedation safety checklist. We hypothesized that implementation of a checklist would improve compliance of critical safety elements (CSEs) (primary outcome variable) and reduce the sedation-related AE rate (secondary outcome variable).
METHODS: This process improvement project was divided into 5 phases: a retrospective analysis to assess variability in capture of CSE within 3 departments that perform deep sedation and the association between noncapture of CSE and AE occurrence (phase 1), design of the checklist and trial in simulation (phase 2), provider education (phase 3), implementation and interim analysis of checklist completion (phase 4), and final analysis of completion and impact on outcome (phase 5).
RESULTS: We demonstrated interdepartmental variability in compliance with CSE completion prechecklist implementation, and we identified elements associated with AEs. Completion of provider education was 100% in all 3 departments. Final analysis showed a checklist completion rate of 75%, and its use significantly improved capture of several critical safety elements. Its use did not significantly reduce AEs (P = .105).
CONCLUSIONS: This study demonstrates that the implementation of a sedation checklist improved process adherence and capture of critical safety elements; however, it failed to show a significant reduction in sedation-related AEs.
Adverse events (AEs) related to delivery of pediatric deep sedation by nonanesthesiologists in high-reliability organizations generally occur at a low frequency but may be associated with high acuity.1,2 In pediatric sedation, national collaboration has increased awareness of this issue.1 Process improvement measures to reduce these events could improve safety during deep sedation. In other industries such as nuclear medicine and aviation, checklists have been used to increase safety.2,3 Similarly, in health care checklists have increased safety, including the World Health Organization (WHO) surgical checklist, which has been shown to reduce surgical mortality and morbidity.4 Checklists have also helped improve adherence to processes of care and provide a framework for team communication.5 Simulation trials have provided the vehicle for successful introduction of checklists into workflows.6,7
Our freestanding children’s hospital performs >4000 sedations annually, (∼2500 deep sedations). The majority of sedations are performed by our Pediatric Hospital Medicine–Based Sedation Service (PHM-SS), followed by the emergency department (ED) and PICU. All deidentified data for sedations performed by the PHM-SS are submitted to the Pediatric Sedation Research Consortium (PSRC) Web-based collaborative database.8,9 Sedation-related AEs are reviewed quarterly by our interdisciplinary Sedation Quality Subcommittee for peer review and quality improvement processes. Our high-volume sedation service with multiple providers and locations offered a unique opportunity to reduce variability and increase safety.
Our institution recently implemented a Procedural Sedation Safety Checklist for the 3 areas in the hospital where deep sedation is performed (PHM-SS, ED, and PICU). The checklist was designed by modification of the WHO surgical checklist and incorporation of critical safety elements (CSEs) as outlined by the American Academy of Pediatrics, the American Society of Anesthesiologists, the Society for Pediatric Sedation, The Joint Commission, and those identified by our sedation leaders via local quality improvement.8,10
We hypothesized that the implementation of the checklist would improve process adherence and compliance of CSEs in all departments performing deep sedation and reduce deep sedation–related AEs.
The PSRC is a national collaborative database and the research arm of the Society for Pediatric Sedation and comprises a collaborative, multi-institutional prospective data collection system of sedation encounters from member institutions.8 The primary endpoint of AE reduction was measured with our institutional data for PHM-SS submitted to PSRC and standardized data fields, definitions, and abstraction of AE rates. ED and PICU sedation data were obtained from the institutional database maintained and reviewed by the Sedation Quality subcommittee.
This process improvement project was divided into 5 phases: a retrospective analysis to assess variability in capture of critical safety elements within 3 departments that perform deep sedation and the association between noncapture of CSE and AE occurrence (phase 1), design of the checklist and trial in simulation (phase 2), provider education (phase 3), implementation and interim analysis of checklist completion (phase 4), and final analysis of checklist completion and impact on outcome after 5 months of checklist use (phase 5).
Phase 1: Assessment of Current State (Prechecklist)
This phase was designed to assess:
The degree of interdepartmental variability in process implementation and documentation of sedation-related CSEs. Department nurse managers performed a retrospective convenience sample chart audit of 50 charts from each of the 3 departments that perform deep sedation and collected data on CSE completion and sedation location. This sample measured only variability in capture of prechecklist CSEs and was used to compare postchecklist CSE completion in these 3 areas.
The relationship between incomplete CSEs and occurrence of AEs. This was assessed via another independent retrospective randomized comparison of all deeply sedated patients in the year preceding the checklist creation that had a moderate to severe AE (defined as “cases”) with all patients (stratified by randomly selected dates of procedure [3 dates/month] in an ∼1:9 case–control ratio) who received deep sedation but did not have an AE (defined as “controls”). A blinded expert (quality director of sedation services) also determined accuracy of American Society of Anesthesiologists (ASA) allocation retrospectively in this data set.
Moderate AEs were defined as laryngospasm, airway obstruction, apnea, nonminimal hypoxia, aborted procedure, prolonged recovery, seizure, stridor, unexpected change in vital signs by ≥30% from baseline, agitation or delirium, or the use of a reversal agent or bag valve mask ventilation. Severe events were defined as anaphylaxis, aspiration, cardiac arrest, emergent anesthesiology or PICU consult, code blue or medical response team alert, unplanned intubation, escalation in level of care (unplanned admission or PICU transfer), or death.9 Specific demographics of cases and controls were compared.
Pearson χ2 or Fisher’s exact test (depending on cell sample size distribution) were used to assess associations between CSE completion rates and occurrence of AEs with calculation of odds ratios. Multivariate logistic regression for CSE was performed to identify independent predictors of reduced AE risk and to model the likelihood of an event as a function of a specific CSE omission.
Phase 2: Checklist Creation and Testing Via Simulation
A 2-page checklist was designed by modifying the validated WHO surgical checklist to the sedation workflow (Supplemental Fig 3). The sedation team as specified in the checklist consisted of the sedation nurse, the sedating physician, and the procedure technician or physician. The checklist was divided into preprocedure, intraprocedure, and postprocedure components. Each component required distraction-free communication between all team members for completion. The checklist additionally included references for the sedation process such as risk stratification criteria to facilitate triage to anesthesia or to identify potential high-risk patients, dosing guidelines, sedation scale, review of discharge criteria, and so on.
Checklist usability was evaluated and team feedback was obtained in the simulation environment by representative sedation teams from each of the 3 departments using the checklist within their unique environments and work flows.
Phase 3: Education of Sedation Providers
All sedation nurses and physicians were required to complete education through online modules consisting of a didactic PowerPoint presentation describing the purpose and elements of the checklist, a simulation video demonstrating its optimal use, and a posttest. Education also occurred in person at staff and department meetings.
Phase 4: Implementation and Compliance Tracking
The checklist was implemented on October 1, 2013 in all 3 departments. At 6 weeks after implementation, an interim audit was performed to assess completion of the checklist. Documentation of CSE before and after implementation were compared via Pearson χ2 tests to determine whether a significant increase occurred in previously deficient elements. This information was shared with the sedation staff at department meetings to provide timely feedback to nurses and physicians on checklist completion.
Phase 5: Final 5-Month Post–Checklist Implementation Analysis
Moderate to severe AE rates were compared in the 5 months before and after the Pediatric Sedation Safety Checklist implementation period via the Pearson χ2 test. Final analysis was also performed to assess completion of checklist CSEs at 5 months.
Phase 1: Prechecklist State
The first goal of phase 1 was to document the degree of variability between the three sedation departments. The prechecklist chart audit convenience sample of 50 charts showed variability in compliance of CSE and its documentation (Table 1). Of the 19 elements audited, weight-associated elements and compliance with institutional dosing guidelines were not documented by any department. Only PHM-SS used an objective risk stratification prescreening tool. The other departments used ASA class and patient history as prescreening criteria.
Phase 2: Checklist Creation and Testing Via Simulation
In a randomized case–control study, 65 patients with AEs in the year preceding the checklist (cases) and 596 patients without AEs (controls) were compared with respect to clinical variables and CSE documentation rates. There was a difference in ASA class, weight >95th percentile, and sedation location between groups (Table 2). There was variability in CSE documentation rates between cases and controls (Table 2).
Univariate and multiple logistic regressions identified the CSEs that predicted AEs (Table 3). Completion of 4 CSEs was independently associated with reduced AEs. Completing the medication double check was associated with a threefold increase in the odds of an AE. Furthermore, 100% compliance with these CSEs predicted an estimated AE rate of 4% according to a backward stepwise elimination model (Table 3). In addition, every instance of dosing medications greater than maximum recommended by institutional guidelines (N = 15) was associated with an AE.
Phase 3: Goal: Education of All Providers (Physicians and Nurses Involved in Deep Sedation Delivery)
Education of providers was achieved with a 100% completion rate of review of the online PowerPoint presentation and video simulation and passing the posttest. Training of all providers took 3 months, and implementation of the checklist occurred within ∼2 months of completion of training.
Phase 4: Goals: Checklist Implementation, 6-Week Postimplementation Interim Audit of Checklist Compliance, and Comparison of Completion of CSE Prechecklist and Postchecklist
An interim audit of checklist completion was performed at 6 weeks after implementation (October 1, 2013–November 10, 2013), with an overall checklist completion rate (defined as completion of all elements on checklist) of 67%. This interim analysis included 232 checklists divided into “completed and compliant” group (N = 156) and “incomplete or noncompliant” group (N = 76). “Completed and compliant” checklists were the ones that had all CSE sections completed ideally (Fig 1A). “Incomplete or noncompliant” checklists were defined as ones in which ≥1 critical elements were left unchecked or incorrect options were checked (ie, “Consent” box was checked but checked as “NO”). Compliance data from interim analysis were shared with individual departments for an opportunity to reeducate and optimize checklist use and completion.
Post–checklist implementation data showed a significant improvement (P < .05) in process implementation with regard to capture of several CSEs in all 3 departments (Fig 2 A–C).
Phase 5: Goal: Final Analysis (5 Months Postchecklist) Was Performed, and Total AE Rate Before and After Checklist Was Compared
Final analysis >5 months after checklist implementation showed a total of 773 deep sedation encounters, 560 with “completed and compliant checklists” and 213 with “incomplete or noncompliant checklists.” The checklist completion rate rose from 67% to 75% after reeducation and individual departmental feedback at the interim audit. The AE rate in the “completed” group was 3.39% (19/560) and in the “incomplete” group was 4.22% (9/213), P = .580 (Fig 1 A and B). The odds of an AE occurring in the “incomplete” group was 1.3 times more likely than the “completed checklist” group (95% confidence interval, 0.559–2.822).
Overall AE rate in the 5 months immediately before checklist implementation was 5.2% (54/1033) and decreased to 3.6% (28/773) at 5 months after implementation (P = .10).
This study is the first pediatric study evaluating the impact of process improvement via a sedation checklist on outcomes. We demonstrated that implementation of the checklist improved process adherence and documentation of critical safety elements; however, it failed to show a significant reduction in sedation-related AEs before and after implementation.
Prechecklist chart audit showed variability in process adherence to CSEs across departments. We also found that omissions of several CSEs were associated with AEs in our case–control analysis. Cases and controls were well matched except in categorization of ASA class and weight >95th percentile, which were significantly higher in the cases. This result was expected based on previous studies confirming higher ASA class and obesity as being risk factors associated with AEs.9 In our study, AEs were also more likely to occur in the PICU and ED than in PHM-SS. This difference probably reflects emergent PICU and ED sedations in patients with higher acuity. Compliance with 5 specific CSEs predicted reduced AE risk (Table 3). Of these elements, medication double check was higher in patients who had an AE, which was not anticipated. Two theories may explain this finding. First, it may reflect variability across departments in policies that mandate medication double check, such as with narcotics. Therefore, we reviewed our data to see whether these cases involved more invasive procedures where narcotics would be used, but the data did not support that correlation. Alternatively, medications may have been double checked after an AE, suggesting that double checking medications occurred as a result of rather than a precursor or predictor of AE. We also found that an inappropriate ASA allocation was an independent variable associated with the occurrence of an AE. Formalized objective risk stratification or prescreening tools might mitigate this risk by decreasing the variability associated with a subjective ASA allocation by providers with varying experience and expertise in sedation.
In our case–control study, adjustment for different risk factors was not performed, and crude odds ratios are presented. The case–control comparison was meant to identify risk factors in our population that we should recognize and therefore include in the sedation checklist or training regimen for its implementation. We cannot control for that effect because we have incorporated it into the study intervention. The effect of location, BMI, and ASA class differences in the 2 groups allowed us to design a better checklist and training program because strata-specific checklist items were incorporated (eg, appropriate ASA documentation and a risk prescreen, patient >95% percentile, BMI calculation, and dosage adjustment for ideal body weight). Because we cannot change who we sedate or restrict it to certain locations, we did not adjust for those risk differences. The only thing we can change for our sedated population is how we recognize risk factors and formulate a sedation plan for the standard and higher-risk patient. The purpose of multivariate model is to quantify the associative effects of factors that we can change, which is CSE capture. The alternative of stratification and performing this modeling within certain strata such as sedation services alone would limit our ability to advocate for performing the checklist for our general population.
Postchecklist process documentation improved for capture of multiple CSEs across all departments (Fig 2). The performance of 4 CSEs that improved the most in all 3 departments (from 0% to 100%) were obesity identification, BMI calculation, dosing weight calculation, and medication dose referencing with institution guidelines. Completion of some of these elements was shown retrospectively in our study to reduce the odds of having an AE (Table 3).
Overall AE rates after checklist implementation did decrease by 30%, but it was not statistically significant. This finding could be due to a low baseline rate of moderate to severe AEs in deep sedation at our high-reliability organization. Thus, the numbers needed to show statistical significance are large and may need to be tracked over a longer time or across multiple institutions. We would need 2383 sedations to detect a statistically significant reduction from the prechecklist AE rate of 5.2% to the 4% predicted postchecklist value, assuming 80% power calculation. Another factor may be that AEs in this study were grouped together among 3 departments that sedate different patient populations using different providers, and the role of checklist use in reduction of AEs in each department could not be studied independently because the study was underpowered. This heterogeneity may have contributed to β error.
Our study has several limitations. It is known that checklist success is reliant on compliance and true adherence to team communication, not just the act of checking boxes.5 Direct observation of sedations is the best way to capture true checklist compliance. This observation was not done in our study. Provider education and team training are also essential to checklist success. Although all providers completed all online education materials and testing, not all completed simulation training.5 Additionally, our chosen period of 5 months postchecklist was probably too short to show a decrease in AEs related to checklist use. When the Veterans Health Administration used their surgical checklist, mortality continued to decrease with each passing quarter. These results are supported in other studies that used a safety checklist, where improved outcomes occurred over an extended period of time.5,11
This study did incorporate several strategies into its design to increase checklist success. Users at our institution were more likely to buy into the Pediatric Sedation Safety Checklist, given that it was designed by physicians and nurses at our institution and modified from the WHO surgical checklist to fit our local culture of safety and our workflow.5,12 Our checklist completion rate was 75%, comparable to that in previous studies.12 Interim analysis supported the need for tracking compliance and usage of the checklist and providing timely and focused feedback with reeducation. We also found that sharing the data supporting the need and value of the checklist to increase safety with the end users increases buy-in and hence the checklist compliance. This finding is consistent with the idea of ownership of the checklist.5,12
We have incorporated the checklist into our electronic medical record and are expanding its use to include an “extended timeout” that includes all the checklist elements and CSEs based on feedback from end users. Continued feedback and involvement from sedation providers may assist in full implementation toward ideal compliance and outcome improvement over time.
A sedation safety checklist can be a valuable tool that increases process adherence and completion of CSEs in pediatric sedation. Its impact outside the pediatric health care arena could expand to adult and community emergency and inpatient units where pediatric sedations are not routinely performed and hence would serve as a visual cue to improve compliance of CSE and interteam communication.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Connors JM,
- Cravero JP,
- Kost S,
- LaViolette D,
- Lowrie L,
- Scherrer PD
- Cravero JP,
- Blike GT,
- Beach M,
- et al
- Scherrer PD,
- Mallory MD,
- Cravero JP,
- Lowrie L,
- Hertzog JH,
- Berkenbosch JW
- Hoffman GM,
- Nowakowski R,
- Troshynski TJ,
- Berens RJ,
- Weisman SJ
- Hand L
- Treadwell JR,
- Lucas S,
- Tsou AY
- Copyright © 2017 by the American Academy of Pediatrics