hosppeds
September 2015, VOLUME5 /ISSUE 9

Comparison of Pediatric Early Warning Score to Physician Opinion for Deteriorating Patients

  1. JB Fenix, MD, MPHa,
  2. Catherine W. Gillespie, MPH, PhDa,b,
  3. Amanda Levin, MDa,b and
  4. Nathan Dean, MDa,b
  1. aChildren’s National Health System, Washington, District of Columbia; and
  2. bGeorge Washington School of Medicine, Washington, District of Columbia
  1. Address correspondence to JB Fenix, 1716 N. Edgemont St, Los Angeles, CA 90027. E-mail: jb.fenix{at}gmail.com

Abstract

BACKGROUND: This study compares a Pediatric Early Warning Score (PEWS) to physician opinion in identifying patients at risk for deterioration.

METHODS: Maximum PEWS recorded during each admission was retrospectively ascertained from electronic medical record data. Physician opinion regarding risk of subsequent deterioration was determined by assignment to an institutional “senior sign-out” (SSO) list that highlights patients whom senior pediatric residents have identified as at risk. Deterioration events were defined as intubation, initiation of high flow nasal cannula, inotropes, noninvasive mechanical ventilation, or aggressive fluid resuscitation within 12 hours of transfer to the PICU. We assessed the relationships of sociodemographic variables, PEWS, and SSO assignment with subsequent deterioration events using multivariate regression analysis to control for a number of covariates.

RESULTS: There were 97 patients with nonelective transfers to the PICU who were eligible for placement on the SSO lists before transfer, 51 of whom experienced qualifying deterioration events. Maximum recorded PEWS was significantly higher for patients with a subsequent deterioration event during the first 12 hours after transfer, compared with those who were transferred but did not experience a deterioration event in the first 12 hours (mean [SD]: 3.9 [2.0] vs 2.9 [2.0]; P = .01). This association persisted even after multivariate adjustment. SSO assignment was only marginally associated with risk of deterioration among this patient population, with or without adjustment for covariates.

CONCLUSIONS: The PEWS was significantly associated with ICU deterioration, whereas physician opinion was not. Used alone or in conjunction with physician assessment, PEWS is a valuable tool for identifying patients vulnerable to acute deterioration.

Early recognition of the deteriorating child is an important goal of inpatient pediatric care,1,2 and evidence suggests that delays in recognition and rapid intervention significantly increase risk of mortality.3 Application of timely critical care services can significantly improve outcomes.46

In 2006, the Institute for Healthcare Improvement identified rapid response systems as a core tool to reduce preventable loss of life,79 with rapid response systems including both detection and response components.10,11 Early warning systems (EWS) are tools that help to identify deteriorating patients by combining nursing assessments and objective vital signs into a composite score with associated escalation policies mandating increased frequency of assessments or interventions when scores are elevated. Variations of EWS are now commonly used at major adult and pediatric medical centers.12,13

These EWS are well validated,1418 increase identification of deteriorating children,19 and supplement improvements to patient safety.20 EWS can be combined with rapid response teams (RRT) that decrease cardiac arrest2123 and significantly decrease mortality2427 in both prospective studies28 and across multiple settings.2,2931 However, the specific scoring algorithms vary among EWS such that existing data reflect a wide range of sensitivities, specificities, and predictive values.32,33 The metric to which EWS are compared is also changing. Although ICU transfer or RRT/Code activation are classically used as end points of clinically significant deterioration, a recently developed measure of deterioration comprising intubation, positive airway pressure, and/or vasopressor administration has been suggested as a pragmatic proximate measure for mortality and arrest given that these are relatively rare events in the pediatric population.34 Finally, the interplay between pediatric early warning scores (PEWS) and clinical judgment is less well known,35 with some evidence suggesting the superiority of clinical judgment in particular settings.36 The aim of this study was to compare a prospectively validated PEWS tool6,37 to physician opinion in predicting patient deterioration in the PICU.

Methods

We conducted a retrospective study of all patients nonelectively transferred to the PICU and cared for by pediatric residents from August 2012 to April 2013. Patient information was abstracted from the medical record and from the pediatric resident handoff tool. Approval for this study was granted by the medical center’s institutional review board.

Setting

The study was conducted at a large tertiary children’s hospital, with >300 inpatient beds and 44 PICU beds. Our institution initially established a RRT and PEWS protocol 5 years before the study. In summer of 2012, before the initiation of our study, the PEWS in use at our institution was revised to improve the sensitivity of the tool among our patient population and to account for the fact that high flow nasal cannula (HFNC) is restricted to ICUs at our institution. A corresponding tiered algorithm of escalation of care was also revised, and hospital-wide reeducation of these systems was implemented.

Measures of Interest

The outcome of interest was deterioration defined as any intubation, inotropes, HFNC, noninvasive mechanical ventilation, or aggressive fluid resuscitation (>60 mL/kg) within 12 hours of transfer to the PICU. This is a variation of the critical deterioration metric that has previously been validated and associated with in-hospital mortality34 but includes the additional measures of HFNC and aggressive fluid resuscitation. Mortality itself was not used because it was a relatively rare event in the pediatric population over the study period.

The PEWS used in this study is a 3-component score based on age-stratified respiratory rate, heart rate, and behavior as collected and verified by registered pediatric nurses at 4-hour intervals. Composite scores range from 0 (all parameters normal) to 13 (all parameters abnormal). This PEWS is a modification of the Brighton PEWS developed by Monaghan,6 which was prospectively validated by Tucker et al,37 demonstrating good sensitivity and specificity to predict ICU transfer, as well as ease of use compared with instruments with a greater number of components. Cut points of ≥3 and ≥5 were assessed in accordance with previous on-site institutional quality improvement studies and internal policies that activate the RRT.

Physician opinion regarding risk of subsequent deterioration was defined as assignment to institutional senior sign-out (SSO) lists. Similar to other institutions, senior resident physicians are the primary contact for bedside providers, attending physicians, and consultants and are identified as the first contact in a tiered escalation algorithm. Handoffs among senior residents are by an electronic handoff tool, the SSO. Additionally, the SSO list is the primary method of communicating level of concern between daytime, nighttime, cross-covering, and weekend teams. Our study included all senior residents during the study period (∼40 distinct third-year residents). Patients eligible for SSO list are admitted to hospitalist, pulmonology, adolescent, renal, endocrine, gastrointestinal, or neurology teams; hematology-oncology, NICU, surgical, and cardiac admissions are not eligible for SSO. A more detailed, comprehensive sign-out tool is maintained by the pediatric interns, and thus the separate SSO is used by senior residents as a tool to highlight the patients deemed to have some unique feature, worrisome diagnosis, or perceived higher risk for deterioration. The SSO list is most relevant in this study because it is the standard way in which high-risk patients are identified by senior residents and the way in which the level of concern about these patients is passed from one senior resident to another.

Data Collection

All patients transferred to the PICU nonelectively were identified through our electronic medical record and an internal database. Charts were retrospectively reviewed for demographic information, date of admission, date of transfer to the PICU, inpatient team before transfer, maximum PEWS recorded during the entire hospitalization before PICU transfer, and treatments in the PICU defined in the clinical deterioration metric. Thereafter, the SSO lists for each day during the study period were reviewed to determine whether these patients had been listed by the senior resident, thus indicating physician concern of possible deterioration on or before the date of the transfer.

Data Analysis

The relationships between deterioration status and sociodemographic characteristics, length of hospitalization before PICU transfer, source of admission, PEWS scores, and SSO assignment were evaluated using Pearson’s χ2 tests and Fisher’s exact tests for categorical variables and 2-sample t tests and Wilcoxon rank-sum tests for continuous variables. To control for potential confounding effects of age, gender, race/ethnicity, and timing of transfer after admission, we used multivariable Poisson regression with a log link and robust standard errors for all regression models assessing the relationships of PEWS scores and SSO assignment with deterioration events. We then calculated standard performance characteristics (eg, sensitivity and specificity) using McNemar’s χ2 test for paired proportions to facilitate direct comparisons of each risk assessment tool, among a cohort of patients with nonelective PICU transfers who did and did not experience a qualifying deterioration event within 12 hours of transfer. Lastly, we also assessed sensitivity and specificity of these tools when considering all eligible patient days during the 8-month study interval (N = 16 551), regardless of ICU transfer status. All analyses were performed in Stata Version 13.1 (StataCorp, College Station, TX).

Results

During an 8-month interval from mid-August 2012 to mid-April 2013, there were 97 in-patients with nonelective transfers to the PICU who were also eligible for placement on the SSO lists before PICU transfer. Characteristics of this population are shown in Table 1. Of the 97 patients included in the study sample, 51 (53%; 95% confidence interval 42.5–62.7) required intubation, inotropes, HFNC, noninvasive mechanical ventilation, or aggressive fluid hydration within 12 hours of their transfer to the PICU.

TABLE 1

Sample Characteristics

Bivariate Analyses

In this sample, there were no differences in age, gender, or race/ethnicity with respect to risk of deterioration. Patients who experienced a deterioration event in the 12 hours after ICU transfer had a maximum mean recorded PEWS of 3.9 (SD 2.0) before PICU transfer compared with a maximum mean PEWS of 2.9 (2.0) in those patients who did not experience a deterioration event (P = .01). Patients who experienced a deterioration event within 12 hours of transfer to the PICU had been assigned to SSO 43% of the time, whereas patients without deterioration event were assigned to SSO 30% of the time; this difference was not statistically significant (P = .2).

Table 2 displays the unadjusted relative risks (and 95% confidence interval) associated with SSO assignment or 3 alternative specifications of the PEWS metric (PEWS ≥5, PEWS ≥3, and PEWS as a continuous measure) and patient deterioration. PEWS ≥3 performs similarly to the continuous PEWS metric in identifying patients at risk for subsequent deterioration, and better than either PEWS ≥5 or the SSO designation. Patients with a maximum recorded PEWS before PICU transfer of ≥3 were 84% more likely to experience a qualifying deterioration event in the PICU compared with patients with a maximum PEWS of 0 to 2. Each 1-point increase in maximum PEWS recorded before PICU transfer was associated with a 13% increase in risk of a subsequent deterioration event. The significant associations of the continuous PEWS metric and the binary assessment of PEWS ≥3 with risk of a subsequent deterioration event persisted after adjustment for age, gender, race/ethnicity, and whether the PICU transfer occurred within 24 hours of admission (data not shown).

TABLE 2

Unadjusted RR of Deterioration Events Associated With Risk Classifications Assigned Before ICU Transfer

Comparison of Performance of PEWS and Physician Opinion as Predictors of Deterioration Events After ICU Transfer

Results from a comparison of standard performance characteristics such as sensitivity and percent correctly classified are consistent with results from the multivariable regression models, providing further support for the use of the PEWS cutoff of ≥3 to identify patients at risk for deterioration. Compared with the other risk assessment metrics retrospectively analyzed among our cohort of patients with nonelective PICU transfers, this metric maximizes the percent of patients correctly classified into risk groups (63%; Table 3). A direct comparison of matched proportions indicates that this value is higher, although not significantly so, than the percent correctly classified by SSO assignment (56%; P = .30; McNemar's χ2 test for paired proportions). Creation of a composite indicator by which a patient is considered to be at risk for deterioration if the PEWS value exceeds the specified cutpoint or the patient is placed on the SSO list yields a small improvement in sensitivity (86% vs 80%) but does not improve the overall percent of patients correctly classified (63% vs 62%).

TABLE 3

Performance Characteristics of PEWS and SSO

When we consider all patient days during the 8-month study interval (N = 16 551) and retrospectively assess the performance of PEWS and SSO assignment as predictors of subsequent PICU-level deterioration events among all admissions, PEWS ≥3 was approximately twice as sensitive as PEWS ≥5 and SSO assignment (sensitivity = 80%, 35%, and 43%, respectively). All 3 screening metrics were highly specific in this context (specificity = 94%, 99%, and 93%, respectively).

Discussion

This study is the first to investigate the relationship of physician opinion using SSO lists and PEWS to PICU deterioration events. Although the percent correctly classified by PEWS ≥3 and SSO assignment were not significantly different, only PEWS was significantly associated with deterioration in this setting. The relationship between SSO and deterioration status was marginal. Because PEWS is tied to the escalation algorithm and this involves notification of senior residents, it might be expected that the SSO list would have optimal sensitivity by capturing all of the children with high PEWS in addition to another set deemed sick by a more qualitative physician assessment. However, this was not the case; both SSO and PEWS ≥5 had relatively low sensitivity. This is concerning because it suggests a cohort of children at significant risk of deterioration that are not being captured by one of the most fundamental tools of physician assessment and communication. We had also thought that adjusting for time elapsed between admission and PICU transfer might affect the performance of physician opinion as the care team becomes more familiar with the subtle nuances of individual patients, but this variable was not independently associated with deterioration, and inclusion of this variable in regression models had no impact on the relationship between assigned risk assessment values and subsequent deterioration status. Our results suggest that PEWS should be further integrated into physician communication to enhance the identification of deteriorating patients.

Our study builds on the existing literature in several ways. Our study is not consistent with the existing comparison of physician opinion to PEWS found by Sweney et al.36 This may be due to different measures used for PEWS and/or physician clinical judgment and helps identify an area rich for additional research. As Bonafide et al38 suggest, even score failures may provide a rich field for quality, communication, and performance improvement. Furthermore, although our study does not use the critical deterioration metric defined by Bonafide et al,34 it uses a similar measure that incorporates ICU transfer plus ICU-level interventions instead of ICU transfer only as has been used in many previous studies.16,17,29,37 Further studies are needed to elucidate the relationship of this measure with morbidity and mortality. Finally, although automated trigger systems are becoming increasingly discussed in the adult literature,40 there is a relative paucity of data stemming from pediatric studies. Our study further highlights the potential of automated systems to identify patients who may otherwise be missed by conventional methods.

This study has several limitations. First, it uses SSO lists as a proxy for physician judgment regarding risk for deterioration. As in many institutions, sign-out lists often communicate a larger set of physician concerns, only 1 of which is risk of deterioration. However, we would expect this to result in increased sensitivity and decreased specificity of SSO assignment, which was not found to be the case in this study. Second, our measure of physician judgment is based on the assessment of third-year senior residents whose judgment might be expected to improve with additional experience. However, given the role that senior resident physicians fill as the communication hub for pediatric patients, resident education and training should continue to focus on minimizing errors in judgment and safety. Our study excluded NICU, surgical, cardiac, and hematology-oncology patients. Further investigation is needed to better understand these populations. Finally, our study was conducted at a single center with a limited sample size and over a limited time period. Additional studies at other institutions are needed to further generalize our results.

Conclusions

This analysis of the utility of our PEWS, a derivative of a previously validated tool, among patients transferred to the PICU, demonstrates that the maximum PEWS before transfer is significantly associated with subsequent deterioration, whereas the relationship between physician opinion and subsequent deterioration was only marginal. PEWS ≥3 is more sensitive but less specific than SSO and leads to a nonsignificant increase in the percent of patients correctly classified in terms of subsequent risk of deterioration. Used alone or in conjunction with physician assessment, PEWS is a valuable tool for identifying patients vulnerable to acute deterioration.

Footnotes

  • Dr Fenix helped conceptualize the study design, drafted the initial manuscript, coordinated data collection, and reviewed and revised the manuscript; Dr Gillespie carried out all analyses and reviewed and revised the manuscript; Dr Levin conceptualized and designed the study and reviewed and revised the manuscript; Dr Dean conceptualized and designed the study, designed the data collection instruments, coordinated and supervised data collection, and reviewed and revised the manuscript; and all authors approved the final manuscript as submitted.

  • FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

  • FUNDING: Supported by Award Number UL1TR000075 from the National Institutes of Health National Center for Research Resources. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Center for Research Resources or the National Institutes of Health. Additional funding for statistical analysis was made available by a grant from Ikaria Corporation. Funded by the National Institutes of Health (NIH).

  • POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

EWS
early warning systems
HFNC
high flow nasal cannula
PEWS
pediatric early warning score
RRT
rapid response team
SSO
senior sign-out

References