Validity of Respiratory Scores in Bronchiolitis
Objective: The primary objective of this study was to establish the validity and reliability of 2 respiratory scores, the Respiratory Distress Assessment Instrument (RDAI) and the Children’s Hospital of Wisconsin Respiratory Score (CHWRS), in bronchiolitis. A secondary objective was to identify the respiratory score components that most determine overall respiratory status.
Methods: This was a prospective cohort study in infants aged <1 year seen at Children’s Hospital of Wisconsin for bronchiolitis. We evaluated: (1) discriminative validity (the score’s ability to discriminate between 2 different outcomes) of the respiratory scores to identify emergency department (ED) disposition by using receiver operating characteristic curves; and (2) construct validity (the score’s ability to measure what it is thought to measure, overall respiratory status) by using length of stay (LOS) as a proxy for disease severity and comparing correlations between changes in respiratory scores and LOS. Interrater reliability was established by using intraclass correlation. The contribution of individual respiratory score components to determine ED disposition was studied by using multivariate logistic regression.
Results: A total of 195 infants were included. The area under the receiver operating characteristic curve was 0.68 for CHWRS versus 0.51 for RDAI in predicting disposition. There was no correlation between initial respiratory scores or change in respiratory scores over the first 24 hours and LOS. Item analysis revealed that oxygen delivery, subcostal retractions, and respiratory rate were independently correlated with ED disposition. The CHWRS was more reliable than the RDAI.
Conclusions: The CHWRS had modest discriminative validity in predicting ED disposition. Neither the CHWRS nor the RDAI had good construct validity. Respiratory rate, oxygen need, and presence of retractions were most useful in predicting ED disposition.
Bronchiolitis is a dynamic illness that requires frequent assessment of respiratory status in inpatient and outpatient settings.1 Respiratory scores are used to assess initial respiratory status, follow trends, and evaluate response to therapeutic interventions. There are several different respiratory scores in existence; they vary greatly in composition, from 2 to 5 components on review of current literature.2 The Respiratory Distress Assessment Instrument (RDAI) is one of the most commonly used respiratory scores3–16 and was developed to assess epinephrine response in wheezing infants and initially used by a small pool of study investigators.13 When used as a marker of change with respiratory rate, it is called the Respiratory Assessment Change Score (RACS). The RDAI contains fewer markers of respiratory status than many other scores (wheezing and retractions; respiratory rate is included when measuring change) (Fig 1) and was initially used to determine change in respiratory status as it related to bronchodilator use.13–16 The RDAI validity was established by comparing retractions with other work of breathing markers such as grunting, nasal flaring, and breath sounds.13
Although the focus of the RDAI is on wheezing and dyspnea alone, the Children’s Hospital of Wisconsin Respiratory Score (CHWRS) has a broader focus. It contains 8 markers of respiratory status, including dyspnea, breath sounds, retractions, heart rate, oxygenation, respiratory rate, activity, and cough ability/secretions (Fig 2) and in addition chest x-ray findings and surgical status as it was initially used as a global respiratory score. It was created by a panel of local clinicians and respiratory therapists (RTs) after reviewing scores in existence. The CHWRS is a more comprehensive instrument than the RDAI, although its validity has not been established.
Among the many varying respiratory scores for bronchiolitis, reliability and validity have not been adequately evaluated. To the best of our knowledge, no previous studies have established interrater reliability among the same large pool of raters who use the score in practice. Establishing the validity of respiratory scores in bronchiolitis can be challenging because there is no gold standard for overall respiratory status. Although the RDAI has been validated as an assessment tool for bronchospasm, its validity constructs may have been too narrow to clearly demonstrate an accurate assessment of respiratory status in infants with bronchiolitis.
Bronchiolitis morbidity is more complex than bronchospasm alone,1 and it is possible that a more comprehensive score may best measure the overall respiratory status. Therefore, more thorough investigation of the performance of clinical respiratory scores in bronchiolitis is needed.1,2
We sought to study the validity and reliability of 2 different types of respiratory scores, the RDAI and the CHWRS, in the practical setting of a large group of RTs. We further sought to identify which respiratory score components were most important in contributing to overall respiratory status.
We performed a prospective cohort study of respiratory score validity in an infant population hospitalized via direct admission or evaluated in the emergency department (ED) at Children’s Hospital of Wisconsin (CHW) with bronchiolitis between November 2007–April 2008 and October 2008–April 2009 (Fig 3). The study was approved by the CHW institutional review board, and written informed consent was obtained from parents or guardians.
Inclusion criteria were as follows: (1) age of 0 to 365 days; (2) clinical evidence of bronchiolitis; and (3) symptoms starting within 7 days of presentation. Research assistants evaluated patient charts for the following terms: upper or lower respiratory tract infection, bronchiolitis, bronchospasm, or first-time wheezing, and evaluated the examination section for wheeze, crackles, and/or retractions. Infants who had cystic fibrosis, congenital heart disease, croup, or pneumonia were excluded, as were those who had a history of asthma, wheezing, or bronchodilator use during a previous illness. The ED research assistants were not present from midnight until 7 am, and charts of infants seen in the ED during this time were not reviewed.
RTs participated in a 30-minute RDAI and CHWRS training session before the start of each respiratory season. It should be noted that CHWRS is the RT “home score” and is used frequently throughout the respiratory season. Before the study, the RTs had not used the RDAI. Both scores contain categories for wheezing and retractions and include respiratory rate; however, the way in which these categories are scored differs (Figs 1 and 2).
For admitted patients, an additional condition was placement in the bronchiolitis treatment protocol. This protocol is an RT-driven pathway for the treatment of bronchiolitis that has been effective in reducing length of stay (LOS) at CHW.17 The protocol uses the CHWRS to determine need and frequency of respiratory treatments.
Infants in the ED were evaluated by RTs who used both the CHWRS and RDAI. The RTs did not participate in ED patient care, and treatment decisions were determined by the ED physician, unaware of the scores. If the patient was admitted and placed in the bronchiolitis protocol, the RTs continued to use the CHWRS and RDAI when performing respiratory assessments. All RT treatment decisions were based on the CHWRS. In addition, patients periodically received CHWRS and RDAI assessments by 2 RTs to establish interrater reliability. These RTs conducted their assessments separately but within 5 minutes of each other.
Outcomes and Analyses
Outcomes of this study included an assessment of the validity, reliability, and short-term responsiveness of the RDAI and CHWRS. Criterion validity, the ability to compare a new measure (in our case, a respiratory score) with a known standard, is not possible because there is no gold standard for overall respiratory status in bronchiolitis. We therefore chose to assess the discriminative and construct validity of the 2 respiratory scores.
Discriminative validity focused on ED disposition because this was believed to be a marker of disease severity. Discriminative validity, the ability of the respiratory score to discriminate between admission and discharge, was determined by constructing receiver operating characteristic (ROC) curves for each respiratory score performed in the ED with admission (yes/no) as the outcome. In addition, an item analysis of the RDAI and CHWRS individual score components was performed. By using multivariate logistic regression, we determined the score components that were more closely related to ED admission.
Construct validity focused on the relationship of hospital LOS and respiratory scores measured at the time of admission, at ~24 hours from admission, and at discharge. Construct validity uses related markers to serve as proxies for a gold standard. We hypothesized that there was a relationship between the pace of improvement, LOS, and the overall respiratory status of the patient. We created the following validity constructs from these variables. First, we assumed that patients whose initial respiratory status was most severe would have a longer LOS. Thus, we determined the Spearman’s rank correlation between respiratory score at admission and LOS. Second, we assumed that infants who had more severe respiratory distress on admission would have the highest initial scores and longest LOS and that all infants should have low scores at discharge. Thus, we determined the correlation between change in respiratory score from admission to discharge and LOS. Third, we assumed that infants whose respiratory status improved more quickly over the first 24 hours of hospitalization would have a shorter LOS. Thus, we determined the correlation between the change in respiratory score in the first 24 hours and LOS.
We also assessed the short-term responsiveness of the respiratory scores. Short-term responsiveness refers to the score’s ability to detect a clinical change in respiratory status over a short period of time (eg, 15 minutes’ postintervention). This is important for the purpose of establishing the short-term response to an intervention, which is a common use for respiratory scores in bronchiolitis and recommended by the American Academy of Pediatrics’ bronchiolitis guidelines.1 Establishing short-term responsiveness in bronchiolitis is challenging because no single intervention reliably changes respiratory status in bronchiolitis (unlike bronchodilators in asthma). If both the RDAI and CHWRS were measuring clinical change accurately and similarly, their assessment of change should correlate. Thus, we determined the correlation of change between the CHWRS and RDAI (RACS) before and after interventions such as suctioning, bronchodilators, and hypertonic saline.
Finally, each score’s interrater reliability was evaluated. The intraclass correlation coefficient was used to determine agreement between 2 RTs performing respiratory scores within 5 minutes of each other.
In our estimates of sample size, we used the construct validity outcomes examining respiratory score correlations with LOS. To detect a difference of 0.2 between correlation coefficients of the CHWRS and RDAI, we calculated that 111 patients would be needed. All analyses were performed by using SPSS version 17 (SPSS Inc, Chicago, IL).
Three hundred sixteen patients in the ED were eligible for the study and 216 were enrolled by ED research assistants. Due to RT time constraints, 151 of the enrolled patients had scores completed in the ED. An additional 44 patients were on the hospital floor (admitted directly to the hospital), creating a study population of 195 patients (Fig 3). From this, 95 patients were discharged from the ED and 100 were admitted and placed in the bronchiolitis treatment protocol. Ninety-eight patients who were admitted had scores completed from admission to discharge. The baseline study population characteristics are found in Table 1.
ROC curves were constructed for each score to determine ability in the ED to predict disposition (Fig 4). The area under the curve for the CHWRS was 0.68 with a cutoff point of 7.5 (scores >7.5 predicting admission), giving a sensitivity of 0.65 and a specificity of 0.65. The area under the curve for the RDAI was 0.51, and no conclusion could be made regarding a cutoff point because it was not predictive of disposition.
We evaluated the association between individual components of the respiratory score and need for hospital admission from the ED by using multivariate logistic regression. The dependent variable was admission to the hospital (yes/no), and the independent variables were the score components. By using the CHWRS, oxygen need was most significantly correlated with hospital admission. The respiratory rate in the CHWRS and subcostal retractions in the RDAI also correlated with admission (Table 2).
There was no statistically significant correlation between respiratory score on admission and LOS with either the RDAI or CHWRS (RDAI: r = 0.04, P = .71; CHWRS: r = 0.05, P = .61). Neither score demonstrated correlation between change over the first 24 hours and LOS (RACS: r = 0.07, P = .67; CHWRS: r = –0.23, P = .06). There was a modest correlation between change in score from admission to discharge and LOS with the RACS (r = 0.27, P = .01), but the correlation was poor with the CHWRS (r = –0.05, P = .62). For short-term responsiveness, there was a mild correlation between the change in the CHWRS and RACS after an intervention (r = 0.39, P = .04).
The intraclass correlation coefficient was significantly different between the 2 scores at the 95% confidence interval (CI) level; the CHWRS was 0.73 (95% CI: 0.60–0.82 [n = 72]) and the RDAI was 0.39 (95% CI: 0.17–0.58 [n = 65]) with a group of 46 RTs.
To the best of our knowledge this study is the first to attempt to rigorously validate the ability of 2 distinct types of respiratory scores to assess overall respiratory status in infants with bronchiolitis. We determined that the more comprehensive score, the CHWRS, is somewhat discriminative of admission from the ED, although the RDAI is not. In particular, respiratory rate, oxygen need, and the presence of subcostal retractions were most predictive of ED disposition. Neither score performed well using LOS as a validity construct. We established that the RACS and CHWRS showed some agreement when assessing respiratory status in response to an intervention. The interrater reliability of the CHWRS in a large pool of RTs was good, whereas the interrater reliability of the RDAI was relatively poor.
Our study suggests that the RDAI may be more limited than the CHWRS in assessing overall respiratory status at a single point in time because only the CHWRS had modest utility in predicting admission. If a score is appropriately identifying children who are more ill with bronchiolitis, one would expect it would discriminate ED admission and discharge. Previous studies have enrolled participants based on initial RDAI8–12 but consideration may need to be given to more comprehensive tools. Given that oxygen need is a dominant factor in admission and that it had the highest odds ratio in our item analysis, its absence in the RDAI may limit its use for this purpose. Alternatively, no score may perform adequately because disposition could be influenced by factors other than disease status alone. Parental preferences, follow-up, dehydration, inability to take oral fluids, patient age, and other issues can determine a provider’s decision to admit. In addition, in our study, the RTs were completing the score, and their perceptions of illness severity as based on the score could differ from those of the physicians who are deciding to admit.
We sought to identify the most important respiratory score components in assessing overall status because expert opinion varies on which components are most important. The CHWRS individual score components of oxygen need, respiratory rate, and the RDAI assessment of subcostal retractions were independently correlated with need for hospital admission. Oxygen need is not part of the RDAI but has been shown to be an important marker of illness severity.18–21 Respiratory rate is part of the RDAI only when used as a marker of change within the RACS. Although both scores contain markers of retractions, it was interesting that only the RDAI’s assessment of subcostal retractions was associated with admission. These results lead us to speculate that perhaps all retractions are not created equal. Perhaps subcostal retractions have particular value in the respiratory assessment of bronchiolitis because they are either more noticeable or more sensitive of lower airway respiratory compromise than intercostal or suprasternal retractions. Similar to our findings, a recent study found low oxygen saturation, high accessory muscle use, and respiratory rate predicted need for major intervention in bronchiolitis.22 Based on our study’s findings, the ideal respiratory score should minimally contain: (1) oxygen need; (2) respiratory rate; and (3) retractions.
Although we did not find that the RDAI or CHWRS was valid by using LOS-based constructs, this could be due to problems with the scores or problems with the validity constructs themselves. Our initial assumptions likely underestimated the impact of items outside of respiratory score on LOS. Previous research has identified influences on LOS in bronchiolitis, including prematurity, feeding ability, oxygen need, environmental and social factors, practice variation, and others.18,23–28 Furthermore, bronchiolitis is a dynamic illness that may worsen after admission, thus confounding the relationship between initial scores and improvement with LOS. The American Academy of Pediatrics’ 2006 diagnosis and management guidelines specifically state “The physical examination reflects the variability in the disease state and may require serial observations over time to fully assess the child’s status.”1 Thus, assessments at a few points in time may not be sufficient.
The CHWRS and RACS correlated mildly in response to an intervention, which suggests that at least they could be measuring similar aspects of respiratory status. However, it does not indicate which score is more accurate at measuring change. Although the American Academy of Pediatrics recommends using objective measures of respiratory status to judge response to interventions such as bronchodilators, evidence for the validity of respiratory scores in this outcome is difficult to establish.
The CHWRS was the RTs’ “home score,” and its better reliability was therefore not surprising. The RTs had not used the RDAI, but wheezing, retractions, and respiratory rate are part of the CHWRS. Although they are measured differently (quantitatively and qualitatively), no additional aspects of respiratory status were evaluated. In contrast to previous studies demonstrating good reliability,11,13,15 we found that the RDAI had relatively poor interrater reliability in our institution. The reliability differences may indicate a need for more thorough training and regular use if a score is to be implemented. In addition, although the CHWRS contains more components and at first glance would seem to take more time to complete, our RTs stated the time required was similar between the 2 scores. Verbal feedback from the RTs indicated that they struggled with the wheezing component of the RDAI, resulting in a slightly prolonged time to complete fewer score components. This feedback further demonstrates the importance of regular score use.
Our study contains some important limitations. First, it is limited to the use of respiratory scores by RTs. Results may be applicable only where respiratory interventions and pathways are RT driven. Second, the CHWRS is more familiar and routinely used by RTs at our institution. Third, the CHWRS score determined the patient care decisions and its elements may have been more highly prioritized. Fourth, the item analysis was performed on ED disposition, and caution should be used when generalizing these results to other purposes such as short-term responsiveness. Fifth, LOS and hospital admission may be influenced by items other than respiratory status, limiting the validity constructs’ effectiveness. Sixth, some patients with bronchiolitis were missed either due to RT or research assistant availability. Finally, our study contained slightly fewer patients than the goal for the validity construct outcomes and may have been underpowered; however, correlations were not near significance.
Analysis of the RDAI and CHWRS suggests that the CHWRS more effectively identified infants admitted with bronchiolitis. The greater interrater reliability of the CHWRS demonstrates the importance of training in establishing a reliable score. Both scores were poorly correlated with LOS-based validity constructs. Future efforts toward score improvement should consider using the most important elements of the RDAI and CHWRS (ie, oxygen need, respiratory rate, subcostal retractions) to develop a more effective tool. Validity of respiratory scores should also be studied among the individuals who often make patient care decisions such as ED disposition, hospital disposition, and continuation of various respiratory treatments. Additional study is needed to determine the short-term responsiveness of scores to validate their use in guiding bronchiolitis management.
This work was presented in part at the annual meeting of the Pediatric Academic Societies; Vancouver Canada, May 4, 2010.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by a grant from the Children’s Research Institute and Children’s Hospital and Health System Foundation at Children’s Hospital of Wisconsin.
- confidence interval
- Children’s Hospital of Wisconsin
- Children’s Hospital of Wisconsin Respiratory Score
- emergency department
- length of stay
- Respiratory Assessment Change Score
- Respiratory Distress Assessment Instrument
- receiver operating characteristic
- respiratory therapist
- 1.↵American Academy of Pediatrics Subcommittee on Diagnosis and Management of Bronchiolitis. Diagnosis and management of bronchiolitis. Pediatrics. 2006;118(4):1774–1793.
- Corneli HM,
- Zorc JJ,
- Mahajan P,
- et al
- Lowell DI,
- Lister G,
- Von Koss H,
- McCarthy P
- Unger S,
- Cunningham S
- Zorc JJ,
- Hall CB
- Aujard Y,
- Fauroux B
- Christakis DA,
- Cowan CA,
- Garrison MM,
- Molteni R,
- Marcuse E,
- Zerr DM
- Copyright © 2012 by the American Academy of Pediatrics