Abstract
BACKGROUND AND OBJECTIVES: The Pediatric Early Warning Score (PEWS) aims to improve early recognition of clinical deterioration and is widely used despite lacking evidence of effects on outcome measures such as hospital mortality. In this qualitative study, we aimed to study effects of both PEWS and the locally designed risk stratification system by focusing on professionals’ perception of their performance. We also sought to gain insight into the perceived effects of PEWS and the risk stratification system on patient safety and to unravel the underlying mechanisms.
METHODS: A single-center cross-sectional observational study whereby 16 semistructured interviews were held with selected health care professionals focusing on perceived effects and underlying mechanisms. Interviews were transcribed verbatim and coded without using a predetermined set of themes.
RESULTS: Coding from semistructured interviews demonstrated that perceived value was related to effects on different levels of Endsley and co-workers’ situational awareness (SA) model. PEWS mainly improved level 1 SA, whereas the risk stratification system also seemed to improve levels 2 and 3 SA.
CONCLUSIONS: This study shows clear effects of PEWS on SA among professionals. It also points to the additional value of other risk factor stratification systems to help further improve PEWS functioning.
Timely recognition of clinical deterioration followed by an intervention is of utmost importance to reduce pediatric mortality and morbidity. In recent years, attention has been given to improving the early recognition of seriously ill children by designing scoring systems. One such well-known system is the Pediatric Early Warning Score (PEWS), which is designed to alert professionals of the clinical deterioration of patients by scoring a set of vital parameters.1,2 The best-studied PEWS is probably the Bedside PEWS, although other validated systems also exist.3–5 Researchers have demonstrated the effectiveness of the system to identify patients before clinical deterioration (by 1–2 hours), albeit with rather limited sensitivity and specificity.6 However, recent data from the United Kingdom and the Netherlands illustrates that in clinical practice, many un-validated adaptations of original systems or self-designed systems with unknown performance coexist.7,8 This vast heterogeneity of systems makes it difficult to study the clinical outcomes of PEWS (for example, effects on pediatric in-hospital mortality).
The limited predictive validity of PEWS (because of false-negative PEWS scores) warrants the development of improved warning systems that also include other, more static risk factors with a predictive value for clinical deterioration.9,10 Sometimes, these risk factors are applicable at an early phase (for example, a worried sign [WS]), even when vital signs are still normal. This can subsequently improve functioning of the detection limb of the hospital’s rapid response system. In our opinion, the integration of PEWS with a risk factors signaling system may provide clinicians with a more complete picture of what is going on and may improve their situational awareness (SA) and consequently patient safety. Therefore, inspired by work of colleagues in the United States, in 2014 we implemented the Pediatric Risk Evaluation and Stratification System (PRESS),11 which integrated (1) a subset of predefined risk factors with (2) vital parameters (ie, PEWS) and (3) patient’s responsiveness (alert, verbal, pain, unresponsive [AVPU] score). Predefined risk factors are a “WS” (including family concern), “ICU involvement,” “high-risk treatment” (a potentially risky treatment of which professionals have limited experience), and “transferred patients from general hospitals” (transfers made during on-call hours when staffing is limited).
On the basis of the PEWS, AVPU score, and the coexistence of the above risk factors, all in-ward patients are stratified into risk categories (standard, medium, or high) on admission, with successively increasing levels of care. High-risk patients (so-called “watchers”) receive increased proactive checks from professionals to follow their clinical course more actively (Table 1).
PRESS
The positive effects of PEWS systems on patient safety are not certain, but given the fact that in many western European countries (as well as in Canada and the United States) the vast majority of hospitals use PEWS systems, it seems that (at least intuitively) they are considered useful by professionals. Accordingly, we decided to focus on professionals’ perception of the effects of scoring systems (PEWS and PRESS) on patient safety to reveal useful insights into the underlying mechanisms by which they may contribute to patient safety and hence to legitimize their usage in hospitals.
Our aim was to qualitatively study professionals’ perception of the effects of PEWS on patient safety on clinical wards as well as the additional value of PRESS and provide insights into underlying mechanisms. In doing so, we may provide with this study may provide a framework of relevant factors to consider when implementing (or optimizing) scoring systems.
Methods
Setting
The study site was a university hospital with 3 pediatric wards (72 beds) where specialized care is offered by pediatric surgical and nonsurgical specialists and nurses with standardized pediatric qualifications. Monitoring of normal vital sign parameters as well as various emergency medical interventions can be performed on the wards. A pediatric resident is present at the wards during on-call hours to respond to clinical deterioration, and consultants from different staffs are available on an on-call basis. In the case of critical clinical deterioration, patients can be transferred to a 12-bed, full-facility PICU.
The locally used PEWS, a modified version of the Bedside PEWS, was previously implemented and validated (November 2011).12 All admitted patients are scored by using PEWS 3 times a day or more frequently when scores are elevated and less frequently (but at least once a day) when patients are at standard risk and have stable vital signs for at least 24 hours, to reduce work load. The PRESS was implemented in April 2014. Both scoring systems are fully integrated in the hospital’s electronic health record system (Epic Systems Corp, Verona, WI). Whereas PEWS is calculated automatically based on the patient’s vital data, PRESS status is updated manually in the system at least once a day for each patient by the attending physician. Both the PEWS score and PRESS category are simplified into smiley faces (green, orange, or red) depicted on the electronic health record system. This creates a live dashboard, which facilitates professionals to quickly identify high-risk patients in the hospital. (Fig 1)
Visualization of PRESS in the electronic health record system (adapted and translated into English to improve readability). Depicted are the statuses of 3 patients on a ward. Relevant data are directly shown without having to open the patient’s medical record, creating a real-time dashboard of the ward. PRESS status and last recorded PEWS are depicted by a colored smiley (red = high-risk, PRESS or PEWS score elevated greater than the alarming threshold; orange = medium-risk, PRESS or PEWS score elevated but less than the alarming threshold; green = standard-risk, PRESS or PEWS score normal). Because PRESS status is updated manually (at least once a day during rounds), but vital parameters (and thus PEWS) are often checked more frequently, the most recent PEWS is automatically uploaded to the dashboard (“Last recorded PEWS”). This way, a changing PEWS score can be detected irrespective of the PRESS score (and corresponding PEWS used) that was previously assigned to a patient. The way the PRESS category was calculated is shown in the PEWS/AVPU/PRESS section. Shown are the PEWS and AVPU scores (at the time the PRESS category was calculated), as well as applying risk factors. This way, professionals can interpret the different components of the risk category and frame the clinical context of the patient more completely. For patient A, one can directly see that although PEWS (1) and AVPU (A) are normal, there is a deviant clinical course for which the WS has been activated. This results in the high-risk PRESS, depicted by a red smiley. For patient B, PEWS (1) and AVPU (A) are normal and there are no risk factors active. This results in a standard-risk PRESS, depicted by a green smiley. For both patients A and B, their most recent PEWS scores have remained within the normal range (as depicted by a green smiley). Patient C has a strongly elevated PEWS (13; alarming threshold ≥8), normal AVPU (A), and staff from the PICU are involved in care for the patient at the ward. This results in a High Risk PRESS (because of both alarming PEWS and the PICU risk factor). The most recent PEWS is still alarming (as depicted by a red smiley).
From September 2014 to September 2015, 62.3% of all hospitalized children (∼3000 admissions) were scored by using PRESS, with an increasing tendency for evaluation with PRESS over time but nevertheless reflecting suboptimal implementation in some patient groups (for example, day care and short-stay admissions). Meanwhile, PEWS were calculated for at least 62.3% of all admitted patients (otherwise a PRESS category could not be calculated).
Study Design
This is a single-center cross-sectional observational study. Semistructured interviews were used to provide insight into perceived value and the responsible underlying mechanisms. Purposive sampling was used to recruit professionals who were expected to have had sufficient positive and negative experiences with these systems and could therefore criticize its performance. To achieve sufficient data saturation, scheduled interviews with ≥8 pediatric nursing staff members, 4 pediatric residents, and 2 pediatric consultants (n = 14) were expected to be necessary. Data saturation was measured during coding and was defined as the point at which no new codes, themes, or perspectives were created when coding interviews from a subgroup.
A preset list of questions addressing different topics was constructed by the study group to impose structure on data collection and improve measurability of data saturation on these topics.13,14 Focus was placed on triggering interviewees to talk about their experiences (both positive and negative) with these systems and to reveal their personal ideas relating to underlying mechanisms. The interviews, lasting between 25 and 60 minutes, were conducted in private rooms at the hospital by 1 researcher (A.V.) and were audio recorded for subsequent analysis with the permission of the participants.
The local ethical committee agreed with this research project.
Data Analysis
Interviews were transcribed verbatim and analyzed with Atlas.ti v7 software (ATLAS ti Scientific Software Development GmbH, Berlin, Germany). Researcher A.V. coded all interviews by using Qualitative Content Analysis.13,15 Six interviews that experienced difficulties in coding were jointly discussed and resolved with another researcher (J.F.) to increase the reliability of the coding. Codes from the remaining interviews were adapted accordingly when applicable, and as a result the interviewer did not experience unresolved coding difficulties.
Results
Semistructured Interviews
The original number of participants was 14; however, because of a lack of data saturation experienced in the consultant subgroup, 2 extra interviews were conducted with consultants. Interviews provided a rich quantity of information relating to all of the previously selected topics in a layered, detailed, and nuanced manner. Many personal beliefs and opinions were supported by actual examples from clinical experience, and substantial homogeneity was demonstrated between interviewees. The 16 interviews resulted in 56 and 37 unique PEWS and PRESS codes, respectively. During the coding process of the final interviews of each subgroup, no new codes, themes, or perspectives emerged, and so researchers experienced sufficient data saturation.
Codes were analyzed to produce frequencies to establish the most important codes. To prevent misleading results by repetitive participants who fixed on a certain theme, codes were also rated by the number of interviews in which they were used. For both PEWS and PRESS, codes were analyzed separately and discussed within the research group, starting with the most influential codes.
Notably, many codes strongly related to effects of these systems on SA. It was thus decided to use the SA model described by Endsley and co-workers, 16 which differentiates 3 levels of SA. Level 1 is the perception of relevant information and represents the gathering and signaling of information. Level 2 is the comprehension and interpretation of that information. Level 3 is the projection of future status by combining level 1 and 2 to anticipate future events. Communication is an exceeding factor and end result of the achieved SA.16
Positive effects of safety systems such as PEWS and PRESS on SA have been described in literature.10 Accordingly, this concept of SA was used as a framework to merge codes into main themes.
To study the additional value of risk factor systems such as PRESS on top of PEWS, findings will be discussed separately for PEWS (Table 2) and PRESS (Table 3).
Results From the Semistructured Interviews Relating to PEWS
Results From the Semistructured Interviews Relating to PRESS
PEWS
Level 1: Perception of Relevant Information
At the ward level, participants indicated the use of PEWS to obtain a bird’s-eye view over admitted patients, usually at the start of their shift or during moments of patient handover. The use of smiley faces to indicate the up-to-date PEWS scoring category at all times was mentioned as a crucial part of the system.
On the individual patient level, having predefined normal values for vital signs helps professionals to quickly detect deviant values. By simplifying values into a score, the professional is alerted that something may be wrong because high scores easily draw their attention. Subsequently, trends in (high) scores are used to follow-up on the clinical condition of patients.
Level 2: Interpretation of Relevant Information
An alarming PEWS score triggers professionals to evaluate underlying causes by analyzing separate PEWS scoring components (such as airway, breathing, and circulation).
Level 3: Projection of Future Status
No relevant codes were detected to indicate effects on this level.
Communication
Communication between nurses and residents is facilitated by PEWS. High scores or declining trends are actively used by nurses to alert residents. Nurses state that they feel more confident to contact doctors and demand action when their gut feeling is confirmed by an alarming PEWS score. When the PEWS of a patient increases over time but remains below the alarm threshold, nurses consult colleagues for help in deciding whether they should contact a resident.
PRESS
Level 1: Perception of Relevant Information
The way that risk factors are integrated into a score that is visualized by a smiley face is considered an essential part of the system. The attention of professionals is almost automatically drawn to patients within a high-risk category. The system provides a clear report of the status of all admitted patients and applicable risk factors. Medical staff on call use the system to check wards for high-risk patients (watchers) who may not have previously come to their attention.
Level 2: Interpretation of Relevant Information
The high-risk status of a patient triggers alertness among professionals. Risk evaluation can be performed more thoroughly because applicable risk factors are stated on the ward reports in the electronic health record system, and interviewees stated that as a result they can consider their relevance more easily to changes in the actual clinical condition of the patient.
Furthermore, nursing staff indicated that the proactive “nurse to attending physician status updates” (every 8 hours) increased their perceived feeling of patient safety and team performance.
Management staff also use the PRESS status of patients on a ward at an organizational level to measure the intensity of care on a ward and level of work pressure among hospital staff.
Level 3: Projection of Future Status
Medical staff members stated that assigning a PRESS status to their patient forces them to proactively consider the applicable risk factors. On the basis of all the available information, they decide whether risks are increased and act accordingly. Therefore, these risk factors are used to warn other health care professionals that the patient in question has an increased risk of developing a serious safety event.
Particularly valuable risk factors mentioned include a WS, PICU or NICU involvement, and transferred patients from general hospitals during on-call hours.
During the period of September 2014 to December 2015, 108 of the 135 (80%) WSs were activated in patients with a nonalarming PEWS score at that time. The discrepancy between WS status of these patients and their PEWS score points to usage of WS in an early phase. The WS component “parental concern” is highly valued by participants who stated that it reminds them that parents’ worries can help to identify clinical deterioration and should therefore be addressed accordingly.
Denoting patients who have been transferred from general hospitals or the PICU or NICU as high risk alerts staff to the origin of these patients. This helps to quickly frame the clinical context when dealing with them, especially during on-call hours, and improves their SA. Furthermore, residents stated that the “transferred of patients from general hospitals during on-call hours” risk factor increases the quality of care as a result of a mandatory clinical evaluation of patients by consultants after transfer.
Communication
The use of PRESS has encouraged reporting on all patients who may be at risk for safety events, even if their clinical condition is stable and no interventions have been needed in preceding hours. Reporting all watchers is regarded as an asset of the system because participants stated that their awareness of high-risk patients on the wards increased in comparison with the situation before the introduction of PRESS, when these patients were not automatically mentioned during all handovers.
Among nursing staff, a high-risk PRESS status functions as a trigger to proactively monitor a patient’s condition. The bedside nurse-to-nurse handover results in a more complete view of the current condition and context of the patient at the start of the shift and further increases awareness.
Discussion
Key Findings
The perceived value of scoring systems found here seems to relate to effects on SA. In particular, PEWS affects level 1 via facilitation of the perception that something is wrong, whereas PRESS influences levels 1 and 2 by providing greater comprehension of the situation and additionally encourages professionals to think about risk factors. Furthermore, by assigning a PRESS risk category, professionals are forced to consider possible future deviant clinical courses in patients and anticipate hereupon (for example, by stating “high-risk therapy” or “WS”). This anticipatory behavior of professionals indicates the level 3 SA effects of PRESS and is particularly associated with the WS: it is frequently activated in patients with a normal PEWS score, indicating its use in an early phase and independently from PEWS.
Communication is an exceeding SA factor. The PEWS mainly influences interdisciplinary communication between nurses and residents, whereas the PRESS has a major influence on communication among medical staff. This likely corresponds with the influence on different SA levels found in both systems. The interpretation of relevant information and projection of future status is mainly performed by medical staff, for whom PRESS may thus be more useful than PEWS alone.
However, the finding that nurses use PEWS to communicate with residents and demand for a checkup of the patient also has a disadvantage: because of the limited sensitivity of PEWS, the patient who scores a false-negative may not receive adequate care in time. This makes the case for the introduction of additional risk factors (such as WS) in scoring systems.
The findings in this study indicate that a PEWS system by itself is unlikely to be sufficient to optimally improve SA because it only affects level 1 (perceiving relevant information) and, to a lesser extent, level 2 of the SA model. Proactively signaling risk factors, as seen in PRESS, helps to frame the clinical context and forthwith improves the ability to project the future status of patients, thus raising the SA of those on call to the next level. This represents the actual added value of a risk factor system such as PRESS.
Furthermore, this indicates that aspects other than statistical PEWS aspects (ie, hard data) should also be considered when designing these systems. The added value found here seems to relate to the effects of these systems on the SA of professionals. In addition, it seems that local implementation factors, such as design or software used (in this case, smiley faces to facilitate communication), preconditions (such as agreements on [de-]escalation), and human factors (such as communication) influence the system’s actual performance in reaching SA goals.
The advantage of studying scoring systems by using qualitative methods as performed here is that large multicenter studies are not required. This study could act as a model for an alternative validation of PEWS and other scoring systems, such as PRESS.
Limitations
Some limitations have to be acknowledged. First, this is a single-center study and therefore results may be difficult to generalize. Moreover, no before and after data were used, which reduces the level of evidence.
Second, local determinants acting on the SA of professionals may exist (such as training in communication and crew resource management) and influence their opinions of the value of scoring systems. However, during the study period such training had only been received by the minority of the medical staff, limiting its influence.
Third, there is no quantitative data to back up the findings or to demonstrate increased patient safety after the implementation of these systems. Furthermore, despite rather high scores of the perceived value of these systems, their actual usage is suboptimal; they are used adequately in approximately two-thirds of all admitted patients. This may reflect a gap between the indicated value and actual professional practice. Consequently, it is important to further study the contextual factors that act as barriers and hinder the optimal usage of scoring systems.
Conclusions
With this qualitative study, we demonstrate clear effects of early warning scoring systems on different levels of SA, in particular when integrating vital signs (PEWS) with other risk factors and thereby creating a system that can proactively signal watcher patients (even before the vital signs alarm). We offer a better understanding of the reasons why health care providers perceive elements of a scoring system to be helpful in the detection and treatment of a deteriorating patient.
Footnotes
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
References
- Copyright © 2017 by the American Academy of Pediatrics
In this issue
