A Modified Delphi Study to Identify Factors Associated With Clinical Deterioration in Hospitalized Children
Abstract
OBJECTIVE: Hospitalized children who are admitted to the inpatient ward can deteriorate and require unplanned transfer to the PICU. Studies designed to validate early warning scoring systems have focused mainly on abnormalities in vital signs in patients admitted to the inpatient ward. The objective of this study was to determine the patient and system factors that experienced clinicians think are associated with progression to critical illness in hospitalized children.
METHODS: We conducted a modified Delphi study with 3 iterations, administered electronically. The expert panel consisted of 11 physician and nonphysician health care providers from hospitals in Canada and the United States.
RESULTS: Consensus was reached that 21 of the 57 factors presented are associated with clinical deterioration in hospitalized children. The final list of variables includes patient characteristics, signs and symptoms in the emergency department, emergency department management, and system factors.
CONCLUSIONS: We generated a list of variables that can be used in future prospective studies to determine if they are predictors of clinical deterioration on the inpatient ward.
Hospitalized children who require unplanned transfer from the general inpatient ward to the PICU have significantly higher mortality rates than children admitted directly from the emergency department (ED), even after adjusting for severity-of-illness on admission.1,2 Studies describing the characteristics of hospitalized children who experience clinical deterioration on the inpatient ward have identified risk factors such as age <1 year,3–5 presence of a preexisting chronic disease,3–6 and admission with an acute respiratory illness.5–7
Although there are several early warning scoring systems that may help identify hospitalized children at high risk of deterioration,4,8–13 none are validated for use in the ED to identify children at high risk on admission to hospital.14–17 Previous research suggests that there is a benefit to early identification and follow-up of children at high risk admitted to the inpatient ward. Kotsakis et al published the first study describing a unique follow-up program of all patients discharged from the PICU using a medical emergency team (MET) in 4 hospitals in Ontario, Canada.18 In this multicenter study, the MET conducted planned follow-up visits on all patients transferred from the PICU to the inpatient ward for 48 hours after PICU discharge. This was associated with a small reduction in PICU mortality for patients requiring PICU readmission, suggesting a protective effect of follow-up by the MET for high-risk patients.18 Furthermore, a recent single-center study describing a pediatric MET follow-up program demonstrated a significant reduction in the rate of early unplanned readmissions to PICU after implementation.19 In addition to high-risk patients recently discharged from the PICU, it is possible that there are other children at high risk who would benefit from planned follow-up by a MET. Identification of patients at risk for clinical deterioration at the time of hospital admission from the ED may allow for cost-effective utilization of the MET to improve patient safety.
Ideally, the study of a quality improvement intervention such as planned follow-up by the MET should be accompanied by a reliable tool that can help guide clinicians in selecting patients who are at highest risk of clinical deterioration and may receive the greatest benefit. At the very least, recommendations based on group consensus are of value.
Expert opinion is often sought using a Delphi process to successfully generate lists of research priorities and quality metrics that have the potential to influence clinical practice. For example, the Choosing Wisely campaign recruited a working group of pediatric hospitalists to identify areas of wasted health care expenditures and prioritize 5 cost-saving recommendations.20 The resultant changes in practice may positively affect clinical outcomes for patients.21–23 Similarly, the Delphi technique can be used to achieve consensus in other areas, such as the identification of risk factors for specific clinical outcomes.24,25 As such, it may be helpful in generating a list of potential predictors for deterioration in hospitalized children that can be used in future clinical trials in this area.
The primary goal of this study was to determine the patient and system factors that experienced clinicians think are associated with progression to critical illness in hospitalized children and explore additional factors that may be relevant but have not yet been examined in previous studies. Furthermore, this list of variables can be used in future studies to develop a scoring system to identify children at high risk at the time of their hospital admission from the ED.
Methods
Study Design
We conducted a modified Delphi study administered electronically over a 3-month period in 2015. Similar to the technique described by Gunz et al, participants were asked to suggest items as well as evaluate items presented using a 4-point Likert scale in a series of 3 iterations.26 Ethics approval for our study was obtained from the Health Research Ethics Authority of Newfoundland and Labrador.
Expert Panel
We invited physician and nonphysician health care providers to participate in the consensus process. We generated the initial list of invitees by compiling a list of corresponding authors of key studies in the published literature. We also used a snowball sampling technique, asking invitees to identify other informed individuals, who were then invited to participate. No minimum or maximum number of participants was established a priori.
Procedure
We facilitated 3 Delphi iterations, distributed electronically using REDCap electronic data capture tools.27 In addition to the initial invitation, invitees received 2 reminders to complete each iteration, unless they indicated in the previous iteration that they did not wish to be contacted for subsequent iterations. Consent was implied by completion of each iteration of survey.
In the first iteration, participants were introduced to the concept of clinical deterioration to critical illness in hospitalized children. This was defined as children requiring assistance from a rapid response team or MET, consultation by the PICU, unplanned transfer from the inpatient ward to the PICU, cardiopulmonary arrest outside the PICU, or unexpected death outside the PICU.
After completing a short demographic survey, participants were presented with a series of 38 statements pertaining to patient and system factors that might place children at high risk of progression to critical illness on the hospital ward that were generated by conducting a systematic review of the existing literature (Kristina Krmpotic, MD, Anna Kempinska, MD, Margaret Sampson, PhD, et al, unpublished data 2015). All patient and system factors identified as significant in studies meeting criteria for full review were presented to participants, who were asked to indicate their opinion regarding each statement, using a 4-point Likert scale (strongly disagree, disagree, agree, or strongly agree). A 4-point Likert scale was chosen to avoid participant neutrality of opinion. Participants could also comment on statements to explain their rationale. They were asked to list any additional factors that might be associated with clinical deterioration in hospitalized children.
Quantitative and qualitative data were exported to Microsoft Office Excel 2007 software (Microsoft Corporation, Bellvue, WA) for analysis. After exporting information, it was electronically stored in encrypted files accessible only to the principal investigator. Descriptive statistics were generated for each factor that was presented. Only statements that reached iteration-specific consensus were advanced directly to the final list of relevant factors or discarded from subsequent iterations. The investigators performed content analysis of qualitative information obtained in each iteration, grouping responses by themes but maintaining as much content and initial wording as possible, to generate additional factors presented and evaluated in subsequent iterations.
In the second and third iterations, participants were provided with the descriptive statistics summarizing the group responses and their individual responses from the previous iteration, giving them the opportunity to revise their judgment based on the consensus. Examples of respondents’ comments from previous iterations were also presented to the group.
Definition of Consensus
We defined consensus, a priori, as 100% agreement (combining “agree” and “strongly agree” vs “disagree” and “strongly disagree”) between participants in the first iteration, and 70% agreement in the third iteration. After completion of the first iteration, we were able to determine the size of our expert panel and defined consensus as 80% agreement for the second iteration, before calculating descriptive statistics for that iteration.
Data Analysis and Factor List Generation
Descriptive statistics were summarized for characteristics of the expert panel (eg, geographic location, clinical discipline), response rates, and consensus rates. Because of the small sample size, statistical differences between rounds were not calculated. Factors with at least 70% agreement in the final iteration were included in the final list of factors. Factors were categorized as patient characteristics, signs and symptoms, ED management, and system factors.
Results
Participant Characteristics
A total of 42 individuals from 28 hospitals in Australia, Canada, the United Kingdom, and the United States were invited to participate. Of these, 14 individuals completed the first Delphi iteration. Response rates were 86% (n = 12) in the second iteration and 79% (n = 11) in the third iteration. Respondents were 8 physicians (pediatric hospitalists, pediatric intensivists) and 3 nonphysicians (nurses, educators) from institutions with moderate to large size PICUs in 5 Canadian hospitals and 3 hospitals in the United States. Characteristics of the expert panel are shown in Table 1.
Characteristics of Expert Panel
Main Results
During the first iteration, there was 100% consensus that it is both important and possible to identify hospitalized children at high risk of progression to critical illness.
In total, 57 factors were presented for evaluation by the expert panel (Appendix A). In addition to the 38 factors we presented, another 19 were suggested by participants. Ten of these were patient factors, and 9 were system factors (Fig 1).
Factor presentation, addition, and removal during each iteration.
In the first iteration, we presented 38 factors, 2 of which achieved consensus. Participants suggested an additional 16 factors (eg, arrhythmia in the ED, lack of senior nursing staff on ward). After consideration of comments from the expert panel, overlapping or similar statements were revised, and a total of 50 statements were presented in the second iteration, including the 2 factors that reached consensus in the first iteration, given that these results might influence respondents’ opinions of similar statements. Consensus was reached on 30 statements; 17 important factors were identified, and 13 were eliminated. In the third iteration, we only presented items that had not reached consensus and new items suggested in the second round, for a total of 26 statements. Further consensus was reached on 13 statements; 6 important factors were identified, and 7 were eliminated.
Ultimately, the expert panel identified 21 factors important in identifying children at high risk of progression to critical illness on the inpatient ward. These were 9 patient characteristics, 7 signs and symptoms, 1 factor pertaining to ED management, and 4 system factors. Consensus rates for each iteration are presented in Table 2.
Factors Associated With Clinical Deterioration in Hospitalized Children
Discussion
To our knowledge, this is the first multidisciplinary study exploring the opinions of clinicians regarding patient and system factors present on admission that could potentially identify children at high risk. Using a modified, electronic Delphi process, the expert panel reached consensus that 23 of 57 factors were associated with clinical deterioration in hospitalized children. The final list of variables included those studied previously, as well as new items for consideration in future studies that could test the clinical utility of these potential predictors.
Patient Characteristics
Infants and younger children, particularly those <12 months of age, have been identified as a vulnerable population by our expert panel and numerous published studies.3–5,7,28 This is consistent with existing literature reporting that the vast majority of children requiring admission to PICU are between birth and 2 years of age.29
Children with complex chronic conditions account for a large proportion of pediatric health care utilization,30 particularly in PICU.31 Definitions of medical complexity may vary between studies but can include the presence of ≥1 chronic conditions that are severe or associated with medical fragility, high projected use of health resources (eg, frequent or prolonged hospitalizations, ongoing involvement of multiple subspecialty services), and assistance from medical technology in the outpatient setting.32 Respondents in our study uniformly indicated that these children are at particularly high risk for clinical deterioration on the inpatient ward. More specifically, the expert panel identified children who require home oxygen or home ventilation as high-risk, with nearly one-third of respondents commenting that children requiring home ventilation must be admitted to the PICU when hospitalized at their institution. In contrast, a retrospective cohort study of ventilator-dependent children admitted to the inpatient ward of a tertiary care pediatric hospital found that the rate of unplanned transfer from the inpatient ward to the PICU was only 2.7 transfers per 1000 patient-days, lower than the general pediatric ward population.33 Many hospitalists are able to safely manage these children outside of the PICU, and this variable warrants further study to determine if these patients would benefit from being followed closely by the MET or another group of critical care trained health care providers, particularly in high-volume centers with specialized teams who often manage chronic and complex patients highly dependent on technology.
Signs and Symptoms in the ED
We presented several signs and symptoms of respiratory distress for evaluation by the expert panel. Only 2 of these variables (ie, hypoxia in the ED, high supplemental oxygen requirement in the ED) were felt to be predictors of progression to critical illness in hospitalized children. In 1 prospective study of children <2 years of age with bronchiolitis, oxygen saturation <85% in room air was found to be predictive of the need for ventilatory support (either invasive or noninvasive) during hospital admission.34 Although previous studies have also identified tachypnea and work of breathing as predictors of the need for critical interventions in young children hospitalized for bronchiolitis,34,35 our expert panel reached consensus that tachypnea in the ED was not useful in identifying children at high risk of deterioration after admission and did not reach consensus on the importance of respiratory distress in identifying children at high risk.
Our expert panel identified persistent tachycardia, arrhythmias, and hypotension in the ED as risk factors associated with clinical deterioration on the inpatient ward. A recently published multicenter retrospective cohort study of 2848 infants with supraventricular tachycardia admitted to tertiary care pediatric hospitals in the United States reported in-hospital recurrence rates of 13%, with 18% of all infants experiencing adverse events related to medication side effects (eg, hypotension, hyperkalemia, hypoglycemia, bradycardia) during their hospital stay.36 Although deterioration secondary to recurrence of an arrhythmia may be unpredictable, it may be important to follow these children more closely given the high rate of preventable adverse events associated with newly initiated treatments. Nonsustained hypotension in the prehospital setting and ED is associated with significant increases of in-hospital mortality in adult patients37 but has not been studied in children.
Although the vast majority of our expert panel felt higher severity of illness on presentation to triage and elevated Pediatric Early Warning Score in the ED were risk factors for clinical deterioration, there is no strong evidence in the published literature to support this.19
ED Management
All of the respondents in our study identified children who required a PICU consult in the ED as high risk for clinical deterioration after admission. The clinical course of patients who are considered for admission to intensive care is rarely reported in the published literature, with no studies in children and only 1 prospective, observational study in adults. In the adult study, nearly 21% of intensive care consultations were denied admission to the ICU, with nearly half of these patients having a subsequent unplanned transfer from the inpatient ward to the ICU within 48 hours of the consultation.38 Perhaps children who are unwell enough in the ED to require a PICU consultation should be followed closely whenever possible in hopes of preventing clinical deterioration or managing it quickly when it does occur.
System Factors
Perhaps the most interesting finding of our study was that the expert panel introduced several system factors that have not previously been studied as contributors to clinical deterioration in hospitalized children. Clinicians indicated that admission to an inpatient ward that does not usually care for children of that age or diagnosis and a lack of senior nursing staff on the inpatient ward are risk factors associated with progression to critical illness. Although it may be difficult to control the variability in inpatient ward bed availability during peak admission times and to address a lack of seniority among allied health care providers, other strategies to improve recognition and management of clinical deterioration in hospitalized children with unfamiliar characteristics exist. Investigators at a single tertiary care pediatric hospital reviewed 80 consecutive unplanned transfers from the inpatient ward to the PICU and identified treatment unfamiliar to the inpatient unit as a risk factor associated with clinical deterioration.39 They subsequently developed and trained nurses to use a situational awareness algorithm to identify patients receiving high risk or unfamiliar medications or therapies, resulting in a 50% reduction in serious safety events in their institution. Another single-center study conducted in the United Kingdom demonstrated a reduction in unplanned transfers from the inpatient ward to the PICU, and a reduction in severity of illness, length of stay, and mortality rates for children who did require transfer, by supplementing clinical practice with weekly in situ simulation team training.40
Our expert panel also reached consensus that lack of senior house staff (ie, minimum postgraduate year 3) places children at risk for clinical deterioration on the inpatient ward. Many tertiary care pediatric hospitals have pediatric critical care medicine fellowship training programs with experienced house staff present in hospital 24/7.41 However, smaller pediatric hospitals affiliated with academic institutions are often staffed by more junior house staff overnight. Although we did not identify any studies regarding the relationship between seniority of house staff and patient outcomes in the published literature, there is recent evidence to suggest that presence of a pediatric intensivist in-house 24/7 results in a significant improvement in rates of survival to hospital discharge after an in-hospital cardiopulmonary arrest.42
Advantages and Limitations
We chose a modified Delphi method for our study because it is an efficient and cost-effective way to obtain large quantities of information from experienced clinicians from a heterogenous population who are geographically separated and may never otherwise converge to provide opinion. It has been shown to have high content validity, and maintaining the anonymity of responses reduces bias related to the pressure to conform to the opinions of highly influential panel members.43 The main disadvantages of this method include the lack of established sample size required for an adequate Delphi process, and the potential that some important variables may not be identified, especially if some experts in the field do not participate in the study.
Our study is limited by the demographic of our respondents. We invited participants from Australia, Canada, the United Kingdom, and the United States but had no respondents from the United Kingdom and only 2 from Australia, both of whom withdrew from the study before the final iteration. This may limit the generalizability of results to North American hospitals, particularly when considering system factors. We also did not prioritize recruitment of clinicians practicing pediatric emergency medicine, with only 1 physician included in our expert panel meeting this description. However, ED physicians are not often involved in the care of patients beyond the ED stay and may not be aware of which children admitted to hospital later deteriorate and require critical care interventions. Although our expert panel is overrepresented by practitioners experienced in pediatric critical care, many were initially identified and invited to participate because of their experience related to MET, the primary focus of which is clinical deterioration in hospitalized patients.
Conclusions
We have identified a list of factors potentially associated with clinical deterioration in hospitalized children. Our expert panel reached consensus on patient characteristics and presenting symptoms that can be studied as predictor variables to identify children at high risk on admission to hospital and system factors that should be considered by hospital administrators to design interventions that improve patient safety.
Planned follow-up of patients at high risk of clinical deterioration is a novel use of the MET that may improve the quality of care for hospitalized children. Rather than relying on individual clinician judgment to identify patients most likely to benefit from such an intervention, we can now make recommendations based on expert opinion. In addition, this information can be used to narrow the list of risk factors for future study as potential predictors of deterioration.
Acknowledgments
We thank all of the clinicians who participated in this study.
Appendix A: Factors Presented for Evaluation
Patient Characteristics
Age <2 weeks
Age <6 months
Age <1 year
Age >10 years
History of previous PICU admission
Recent transfer out of PICU
History of prematurity
History of global developmental delay
History of bone marrow or solid organ transplant
Preexisting upper airway problem
Preexisting cardiac disease
Preexisting seizure disorder
Presence of a preexisting chronic/complex medical condition
Diagnosed with “unusual clinical condition”
Home oxygen
Home ventilation
Tracheostomy
Long-term vascular access
Gastrostomy tube
Prescribed particular classes of medications (eg, opioids)
Not accompanied by parents/usual caregivers
Signs and Symptoms in the ED
Transported to hospital by ambulance
Higher severity of illness on presentation to triage
Elevated Paediatric Early Warning Score
Respiratory distress
Tachypnea
Hypoxia
High supplemental oxygen requirement
Arrhythmia
Severe tachycardia
Hypotension
Hypertension
Acidosis
Hemoglobin <100g/L
ED Management
>3 doses of nebulized medication
Fluid bolus
Partial septic workup
PICU consult
Admission Characteristics
Respiratory condition
Nonsurgical cardiac condition
Neurologic condition
Acute renal disease
Suspected nonaccidental injury
Emergency surgery
Transfer from a referring hospital
System Factors
Admission to a ward that does not usually care for the patient’s condition
After hours admission
High patient-to-nurse ratio
Lack of senior nursing staff on the inpatient ward
Lack of dedicated respiratory therapist for the inpatient ward
Lack of senior resident in-house
Lack of consultant pediatrician in-house
Lack of rapid response team
High volume of admissions within a 24-hour period
Admission when PICU is operating at capacity
Lack of treatment protocols or clinical practice guidelines implemented in the hospital
Footnotes
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
References
- Copyright © 2016 by the American Academy of Pediatrics
