Advertising Disclaimer »

We will not be accepting article comments until November 8, 2021, while our site undergoes major changes. We apologize for the inconvenience.

Research Articles

Development and Validation of a Web-Based Pediatric Readmission Risk Assessment Tool

Thom Taylor, Danielle Altares Sarik and Daria Salyakina
Hospital Pediatrics March 2020, 10 (3) 246-256; DOI: https://doi.org/10.1542/hpeds.2019-0241
Download PDF

Abstract

OBJECTIVES: Accurately predicting and reducing risk of unplanned readmissions (URs) in pediatric care remains difficult. We sought to develop a set of accurate algorithms to predict URs within 3, 7, and 30 days of discharge from inpatient admission that can be used before the patient is discharged from a current hospital stay.

METHODS: We used the Children’s Hospital Association Pediatric Health Information System to identify a large retrospective cohort of 1 111 323 children with 1 321 376 admissions admitted to inpatient care at least once between January 1, 2016, and December 31, 2017. We used gradient boosting trees (XGBoost) to accommodate complex interactions between these predictors.

RESULTS: In the full cohort, 1.6% of patients had at least 1 UR in 3 days, 2.4% had at least 1 UR in 7 days, and 4.4% had at least 1 UR within 30 days. Prediction model discrimination was strongest for URs within 30 days (area under the curve [AUC] = 0.811; 95% confidence interval [CI]: 0.808–0.814) and was nearly identical for UR risk prediction within 3 days (AUC = 0.771; 95% CI: 0.765–0.777) and 7 days (AUC = 0.778; 95% CI: 0.773–0.782), respectively. Using these prediction models, we developed a publicly available pediatric readmission risk scores prediction tool that can be used before or during discharge planning.

CONCLUSIONS: Risk of pediatric UR can be predicted with information known before the patient’s discharge and that is easily extracted in many electronic medical record systems. This information can be used to predict risk of readmission to support hospital-discharge–planning resources.

Avoiding potentially preventable unplanned readmissions (URs) is important in maintaining the quality of care provided to patients, particularly children and their families.1–6 Avoiding URs is also of critical interest to hospitals’ economic well-being to avoid reimbursement penalties.3,7,8 In response to these considerations, many health systems have sought to improve discharge planning processes9–11 in patients who are at risk for otherwise preventable readmissions. This can be done implicitly or explicitly. Explicit risk assessment involves the use of a readmission risk prediction. Yet, multiple reviews12,13 of existing readmission-risk–prediction tools highlight a number of limitations of existing algorithms. Namely, the existing predictive algorithms (PAs) often do not predict readmission risk that well,13 with area under the curve (AUC) estimates being near 0.70 in the vast majority of cases. Moreover, these scoring algorithms often either exclude children altogether12 or otherwise exclude important pediatric subgroups, such as neonates.14 Furthermore, PAs may rely on data elements that are not known until after the patient is discharged.15,16 Also of concern is that some PAs use measures that are derived for adult populations and are inappropriate for pediatrics,17 such as the Charlson Comorbidity Index score.18,19 Time horizons for risk prediction also remain limited, with published pediatric PAs to date having focused solely on 30-day readmission risk.14,20 Yet, URs that happen sooner may be of even more concern to young patients, their families, and the pediatric health care stakeholders who care for them. Evidence suggests that URs that occur soon after discharge are likely the readmissions that are more preventable21,22 and may represent patients who, if identified before discharge, may be easier to intervene on in a hospital setting.

With current limitations of PAs in mind, we sought to derive accurate pediatric-specific UR risk-prediction algorithms for readmissions within 3, 7, and 30 days that rely on information that can be determined before the patient is discharged and that can be readily identified in hospitals’ source electronic medical record (EMR) systems.

Methods

Data Source

We used the Children’s Hospital Association Pediatric Health Information System (PHIS).1,23 This administrative database is devoted to pediatric health care quality assessment and resource use and encompasses 51 pediatric hospitals across all regions of the United States.

Cohort Identification

We identified all patients admitted with at least 1 inpatient or observation encounter between January 1, 2016, and December 31, 2017. Patients between the ages of 0 days and 18 years were included in analyses.

Outcome Derivation

We created indicators for URs within 3, 7, and 30 days of a discharge from a previous admission. To do this, we first identified planned readmissions via integration of 2 approaches to planned readmissions identification. First, we relied on the PHIS database indicator of whether an admission was “elective.” This indicator reflects what individual PHIS-contributing hospitals’ experts deemed to be elective admissions in their source EMRs. This approach is consistent with previously validated EMR readmission algorithms.24 The second method we used to classify an admission as planned in advance was the Pediatric All-Condition Readmission Measure.25,26 The Pediatric All-Condition Readmission Measure considers International Classification of Diseases procedure codes that panels of pediatric care experts have been determined to be nearly always planned in advance.25,26 Any readmission that was not classified as planned by either of these 2 planned-admission classification approaches was considered a UR event.

Predictor Selection

We considered predictors of risk that would be commonly identifiable at any given time during a current admission. Furthermore, we relied on consistent and reliable patient- level predictors contained in the PHIS data model that have been widely used in published pediatric research1,27,26,28,29 and are more fully described below. We note, however, that we excluded a number of unmodifiable demographic variables30 as potential predictors of URs, such as sex, race, and ethnicity. Our clinical partners voiced reluctance to use an algorithm that predicted risk on the basis of sex, race, or other sociodemographic factors because quality-improvement efforts aimed at these likely unmodifiable risk factors would be difficult to implement in children’s hospitals, whether at the unit or hospital level. Related feedback we had is that families may be alienated by discharge-planning risk assessments that declare patients (and their families) to be at high risk of a negative outcome because of a patient’s sex, race, or ethnicity. In addition, preliminary modeling of these variables did not offer consistent prediction relative to the selected clinical predictors. Consequently, the predictive value for future patient cases was of concern to us. We did include 2 key demographics: age of the patient and insurance payer type.30 Pediatric age classifications were aligned with national pediatric standards.31,32 We also included insurance payer type at the current admission.25,33 Because the PHIS is a longitudinal database, each patient’s previous hospitalization record was also evaluated as a predictor of URs. Under the tenet that the best predictor of future behavior is past behavior,34,35 we included indicators of whether a patient had a UR within 3, 7, and 30 days before the current admission. We additionally considered previous health care use by including the number of previous hospitalizations regardless of whether these hospitalizations were elective or unplanned.28,33 For the current stay, we included the admission classification. The types of admissions classified in the PHIS data model33 include newborn admission, emergency department (ED) or urgent care center (UCC) admission, and elective hospital admission. The current length of stay was included in the prediction model,25,28 as were PHIS data model indicators of medical, surgical, or infection-related complications. A reference table for International Classification of Diseases, 10th Revision codes for in-hospital medical, surgical, or infection-related complications is available in the Supplemental Information. We also incorporated the Pediatric Medical Complexity Algorithm (PMCA),36,37 which groups patients as having comorbidities, medical complexities, both, or neither. Finally, we included pediatric complex chronic condition (CCC) indicators,1,25,28,33,38–42 which have shown strong and consistent associations with a range of negative health care outcomes for children.

Prediction Methods

We considered 2 distinct prediction methods to evaluate which option may perform best with the available data. These 2 methods included a random-intercept logistic generalized linear mixed model (GLMM) approach as well as a gradient boosting classification tree approach using the XGBoost43 algorithm. The logistic GLMM offers a straightforward generalized linear prediction method. When developing the GLMM predictions, we evaluated random-slope models as well, but such models did not converge and were highly unstable. Marginal predictions were made from random-intercept logistic GLMM models for performance assessment. The XGBoost algorithm is an ensemble machine learning method44 that fits multiple classification trees (weak learners) to the data to improve the model performance. The advantage of the XGBoost over logistic generalized linear modeling approaches is that it accommodates multiway interactions among patient characteristics while preventing overfitting.45–47

Prediction Model Training

We randomly split the cohort such that 67% of the data were allowed to train prediction models. The remaining 33% (holdout data) were used to test the performance of the prediction models. Because the data were longitudinal (ie, some patients had multiple admissions over time), we avoided “data leakage” by ensuring that all respective patients’ readmissions were in either the training or testing cohort. Fivefold crossvalidation of the training data was used to tune XGBoost prediction models. The logistic GLMM naturally accommodates longitudinal patient-level data.48 For the XGBoost model, we limited data leakage further by bootstrap resampling (b = 100) a random admission from each patient in the training data. Because the XGBoost approach has more tuning parameters to evaluate, we found that a learning rate of 0.1 with a tree depth of 7 (up to a 7-way interaction) fit all 3 outcomes best. Variable importance in XGBoost was evaluated by Information Gain (IG),43 which measures the degree of the reduction in error of prediction with a variable. We then trained a full XGBoost model using the last observation available for each patient to limit patient longitudinal intracorrelation bias.

Model Testing on Holdout Data

We assessed model discrimination using the AUC.49,50 The AUC offers an understanding of whether the pediatric readmission risk score (PERRS) algorithms are able to distinguish between those that are more and less likely to readmit. We compared the discrimination AUCs from the XGBoost and the GLMM models for each of the 3 outcomes using the DeLong Z test.51 We then assessed calibration using calibration curves49,52 for the best-fitting models. Calibration extends the understanding of risk predictions by evaluating whether the predicted risk aligns with the observed risk. We used 100 groupings to offer a conservative graphical estimation of calibration for each outcome relative to the more liberal decile grouping.50,52

Results

We identified a large pediatric cohort of 1 111 323 patients with 1 321 376 admissions. Table 1 contains the demographic profile of patients in the cohort, which was racially and ethnically diverse and commensurate with the United States as a whole. Specifically, 18.8% of patients identified as black, and 4% of the sample identified as being of Asian or Pacific Islander heritage. Across the cohort, 19.5% identified as Hispanic.

View this table:
TABLE 1

PERRS Outcomes and Demographics

Readmission Patterns and Clinical Predictors

Overall, 17 566 patients (1.6%) had at least 1 UR in 3 days, 26 460 patients (2.4%) had at least 1 UR in 7 days, and 39 875 (4.4%) patients had a UR within 30 days. Table 1 presents the distribution of UR outcomes in this cohort. The vast majority (82.3%) of patients did not have a readmission of any kind (elective or unplanned). Of those who were readmitted, 10.8% had 1 readmission of any kind, 3.1% had 2 readmissions of any kind, 1.4% had 3 readmissions of any kind, and <1.0% had a 4 or more readmissions of any kind. The maximum number of observed readmissions of any kind was 24 in the 2-year time period of available data. Also noteworthy is that 923 494 (69.9%) admissions of 796 689 (69.6%) patients in the cohort originated from an ED visit. Table 2 presents the distributions of the remainder of features included in each prediction model.

View this table:
TABLE 2

Cohort Clinical Characteristics

Prediction Model Performance

XGBoost models significantly outperformed GLMMs for URs in 3 days (XGBoost AUC = 0.771 [95% confidence interval (CI): 0.765–0.777]), 7 days (XGBoost AUC = 0.778 [95% CI: 0.773–0.782]), and 30 days (XGBoost AUC = 0.811 [95% CI: 0.808–0.814]; Fig 1). In comparison, GLMM AUCs were significantly lower (3 days AUC = 0.738 [95% CI: 0.732–0.744]; 7 days AUC = 0.749 [95% CI: 0.744–0.754]; 30 days AUC = 0.784 [95% CI: 0.781–0.788]). Consequently, we focus the remainder of the results and discussion on predictions from the better-performing XGBoost models.

FIGURE 1
FIGURE 1

PERRS outcome discrimination performance with holdout test data: AUC. The x-axis is the false-positive rate (1-Specificity), and the y-axis is the true-positive rate (Sensitivity) of the prediction. Separate colors and line types represent predictive AUCs for each of the 3 outcomes regarding holdout test data.

Figure 2 presents the XGBoost model calibrations for each outcome. Although the prediction range is restricted because of the rarity of URs, the calibration curves show that the expected risk from the predicted models aligns with the observed risk in the test data.

FIGURE 2
FIGURE 2

PERRS outcome calibration with holdout test data: risk calibration for each outcome. The x-axis is the mean predicted risk separated into percentiles. The y-axis is the observed mean probability of each outcome within those same percentile categories. The diagonal line represents perfect calibration; the risk prediction should roughly align with the observed risk.

Tree-Ensemble Prediction Model Interpretation

Model interpretation of gradient boosting tree-based models cannot be adequately described with many more familiar effect size measures, such as odds ratios. Instead, IG metrics suggested consistent influence of predictors across all 3 outcomes. Figure 3 presents the IG metrics for each outcome (exact IG metrics are available on request). The number of previous readmissions of any kind, irrespective of planned or unplanned nature, most strongly influenced accurate predictions. The second most important consideration was whether the current admission was through the ED and/or UCC. After these 2 predictors, the influence of different predictors on each outcome diverges slightly. To aid further in gradient boosting model interpretation, we provide a simple reference table of average marginal predictions53 as well as minimum and maximum predicted risks for each predictor and PERRS outcome (see Supplemental Information). This supplement provides an understanding of the average impact of different predictors on each PERRS outcome as well as which factors exhibited the highest predicted risk of URs. The most influential predictor of a future UR was a record of previous readmissions of any kind. Mean predicted risk of a UR within 3 days was <0.01 if a patient had only 1 previous hospital admission. Having >1 previous admission increased the absolute risk of a UR within 3 days to 0.03 with an average relative risk increase of 3.81. The maximum predicted risk for a 3-day UR if a patient had only 1 previous admission to the hospital was only 0.08, whereas having 2 or more admissions had maximum predicted risks of readmission in excess of 0.40. A similar pattern of predicted risks of both 7-day and 30-day URs was seen across the range of patient admission history (Supplemental Information). ED-admission–type risk for 3-day (mean predicted risk = 0.02; maximum predicted risk = 0.53), 7-day (mean predicted risk = 0.02; maximum predicted risk = 0.71), and 30-day (mean predicted risk = 0.05; maximum predicted risk = 0.68) URs was also influential in predicting risk for URs. With regard to medical complexity, 1 particularly striking consideration is that patients who were classified as having both chronic and medically complex conditions had an average risk of 30-day readmission of 0.07, which was >2 times the absolute risk of any other PMCA category for any of the readmission time horizons. The maximum predicted risk for patients who had both chronic and medically complex conditions was 0.68; intuitively, this nearly 10-fold difference in absolute risk in the prediction models highlights the challenges faced by many patients with medically complex and chronic conditions. One other noteworthy finding in the prediction dynamics relates to the length of stay of an admission. Risk of a UR within 3 days of discharge remained low overall at just ≤0.02. Additionally, for lengths of stay of ≥2 days, maximum risk remained relatively low as well, with predicted values at ≤0.23. What is noteworthy, however, is the maximum predicted risk estimate for patients who were discharged with a length of stay of <2 days; for those patients, the maximum estimated risk was nearly twice as high, with the maximum risk being estimated at 0.53. Here, the prediction algorithm may be detecting the offset hospitals may face with some patients between early discharge and later risk for readmission. Other effects of interest can be further referenced in the Supplemental Information. However, we encourage readers to explore the scoring algorithm application below to understand different combinations of risk factors because the nature of XGBoost scoring involves nonlinear interactions among terms that are not well profiled on a linear scale.

FIGURE 3
FIGURE 3

Predictive feature importance to risk predictions: IG effect size measures available for nonparametric gradient boosting trees. Higher IG bars indicate greater influence of the variable on UR risk for each time horizon (3, 7, or 30 days); lower bars indicate the variable has little influence on risk predictions.

Scoring and Implementation of PERRS Algorithms

Commonly, there is a trade-off in model accuracy and ease of interpretation when relying on more recent machine learning techniques.54 Moreover, many published algorithms do not provide a means of easily scoring a patient’s risk. To offset this challenge to modern-day machine learning prediction algorithms, we provide a number of use options for hospital staff who care for children. For an interactive scoring tool, interested users can explore different risks and percentiles of absolute risk in a freely available Web application: https://nchs-public.shinyapps.io/PERRS-app/. Furthermore, if you are interested in implementing the PERRS algorithm in their local health system using the code and scoring rules derived from the XGBoost decision tree algorithms, please contact the authors for further guidance.

Discussion

The ability to make risk predictions for early readmission during the initial hospitalization may help improve quality of life for both the patient and family because it facilitates better planning of care.55–57 To date, the vast majority of readmission-focused PAs have been validated almost exclusively on older adults and geriatric populations and contain predictors that are not relevant for children (eg, Charlson Comorbidity Index score).12 In contrast, the PERRS contributes specifically to the more limited prediction resources available for pediatric-focused care. Our findings suggest it is possible to predict pediatric URs within 3, 7, and 30 days of discharge from a current stay by using a limited number of patient characteristics available to hospital staff before discharge. Additionally, the predictors included to profile a patient’s risk of UR reasonably align with considerations from previous work involving interventions for pediatric readmissions58 and are predictors that hospitals may reasonably consider in tailoring quality-improvement efforts or other intervention strategies to prevent URs.

To date, many pediatric UR prediction algorithm efforts14,20,26,28 may be limited in their utility in real-time clinical care and discharge planning settings for reasons such as limited discrimination of risk, validation in relatively small and geographically limited samples of patients, inability to use these risk PAs before a child’s discharge from a current hospital stay, and inclusion of predictors that are not appropriate for children (eg, dementia or an age ≥50 years in the case of the Charlson score, which is often incorporated into pediatric PAs). The PERRS directly considers the previous limitations of pediatric-oriented UR prediction algorithms. Indeed, discrimination results from our PERRS model for 30-day URs are comparable to recent pediatric-focused PAs that contain a larger number of predictors and also include data elements that are not relevant for children, such as the Charlson Comorbidity Index score.14,17

Our incorporation of pediatric CCCs is also unique. However, even incorporation of appropriate predictors for pediatric populations does not necessarily translate into strong predictive ability. For example a recent study that developed a PA incorporating pediatric CCCs found only a moderate AUC of 0.67 for 30-day UR.14 In contrast to this study, our AUC estimate was >0.80; this is likely partly due to the substantially larger sample size available to us and the machine learning methods we evaluated. Specifically, logistic regression that was additive in nature was evaluated in previous work,14 and our use of XGBoost allowed us to incorporate an ensemble of tree structures that are explicitly interactions among conditions. This contrast in AUCs even when incorporating some of the same predictors, such as previous length of stay and pediatric CCC categories, thereby highlights the importance of the need to consider the complex interactions in conditions and care that affect risk of URs for children.

The PERRS 3- and 7-day readmission risk estimation had slightly attenuated predictive discrimination relative to the 30-day PERRS risk model. However, there is little to compare the PERRS for readmissions in these shorter time horizons. To our knowledge, there are no other accurate published algorithms predicting an individual patient’s risk of URs in time horizons sooner than 30 days. Yet, consideration of the impact of readmission in time horizons sooner than 30 days would seem to be important if researchers are working to improve the quality of care for children and their families at a given hospital. Indeed, evidence suggests that readmissions that occur sooner (eg, within 7 days) are more likely to be preventable than readmissions that occur after 7 days.21,22

Risk Prediction in Discharge Planning

Discharge planning is a complex task that requires time and effort on the part of both caregivers and hospital staff.59–61 Despite relative agreement about the many important components of discharge planning,59 there is significant variation in implementation across different hospital settings.62 To at least some degree, this may be due to strains on available staffing resources. There is clear evidence that when hospital staff resources are limited or strained, risk for adverse events,63–65 severe patient complications,66 and readmission67 all increase. Amid situations of high patient census and potentially strained resources for individual patients during discharge planning, a risk prediction as offered by the PERRS may support additional attention to high-risk patients. Patients flagged as being at higher risk for UR may benefit from enhanced focus on already well-established, evidence-based discharge planning strategies. Such strategies include more tailored patient discharge templates that accommodate a broader range of families from different socioeconomic backgrounds, additional focus on medication reconciliation, and more engaged teach-back of postdischarge care plans.68

Limitations and Strengths

Despite offering better prediction, gradient boosting trees do not offer the same relatively straightforward interpretation metrics as the more familiar logistic regression methods.54 Although the methods used do not provide odds ratio effect size measures, we have sought to provide other metrics, such as average marginal effects and IG metrics. Another limitation is that administrative data are deficient in incorporating many of the psychosocial and socioeconomic factors that may affect risk for readmission (eg, instability in housing, nutrition, and family stressors). Additionally, the PHIS is comprised of data from children’s hospitals around the nation but does not contain data on admissions to nonpediatric care institutions, and it may not be representative of readmission dynamics for children who lack access to a children’s hospital in the United States.

Future Directions

Although the PERRS predicts readmission reasonably well and does so with a reasonably accessible number of predictors, future work is planned to examine the utility of this algorithm in aiding clinicians. We are currently working to design quality-improvement efforts oriented toward implementation science69 to reduce patient risk of URs.

Conclusions

The PERRS is able to predict URs within 3, 7, and 30 days for children being discharged from hospitals that specialize in pediatric care. The scoring algorithm’s reliance on relatively few data elements that are known before a patient’s current hospital admission makes it suitable for use in key aspects of care, such as discharge planning and quality improvement. Finally, the PERRS is likely to be generalizable to a variety of both current and future pediatric populations given the size and geographic range of the cohort used to train and validate the PERRS algorithms.

Acknowledgments

We are grateful for the support and education provided by the Children’s Hospital Association Health Informatics team, particularly David Bertoch, Shawn Reid, Donna Fletcher, Matt Hall, and Patti Duda. We also acknowledge the immense support of Raul Bringas and Morgan Peterman at Nicklaus Children’s Health System’s information technology department. Please note that this work in no way represents the views of Northrop Grumman or its affiliates, although 1 author (Dr Taylor) has an affiliation with the organization. All research efforts occurred on-site at Nicklaus Children’s Health System using only Nicklaus Children’s Health System resources.

Footnotes

  • Deidentified individual participant data will not be made available. The source data (Pediatric Health Information System) is not a publicly available data source.

  • FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

  • FUNDING: Funded by the Florida Organization of Nurse Executives Small Grant Initiative (“Development of a Readmission Risk Assessment Tool for Pediatric Inpatients”; principal investigator: Dr Altares Sarik).

  • POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

References

  1. ↵
    1. Berry JG,
    2. Hall DE,
    3. Kuo DZ,
    4. et al
    . Hospital utilization and characteristics of patients experiencing recurrent readmissions within children’s hospitals. JAMA. 2011;305(7):682–690
    OpenUrlCrossRefPubMed
    1. Hechenbleikner EM,
    2. Makary MA,
    3. Samarov DV,
    4. et al
    . Hospital readmission by method of data collection. J Am Coll Surg. 2013;216(6):1150–1158
    OpenUrlCrossRefPubMed
  3. ↵
    1. Toomey SL,
    2. Peltz A,
    3. Loren S,
    4. et al
    . Potentially preventable 30-day hospital readmissions at a children’s hospital. Pediatrics. 2016;138(2):e20154182
    OpenUrlAbstract/FREE Full Text
    1. Wray J,
    2. Lee K,
    3. Dearmun N,
    4. Franck L
    . Parental anxiety and stress during children’s hospitalisation: the StayClose study. J Child Health Care. 2011;15(3):163–174
    OpenUrlCrossRefPubMed
    1. Franck LS,
    2. Wray J,
    3. Gay C,
    4. Dearmun AK,
    5. Lee K,
    6. Cooper BA
    . Predictors of parent post-traumatic stress symptoms after child hospitalization on general pediatric wards: a prospective cohort study. Int J Nurs Stud. 2015;52(1):10–21
    OpenUrlPubMed
  6. ↵
    1. Hasan Tehrani T,
    2. Haghighi M,
    3. Bazmamoun H
    . Effects of stress on mothers of hospitalized children in a hospital in Iran. Iran J Child Neurol. 2012;6(4):39–45
    OpenUrlPubMed
  7. ↵
    1. Friebel R,
    2. Steventon A
    . The multiple aims of pay-for-performance and the risk of unintended consequences. BMJ Qual Saf. 2016;25(11):827–831
    OpenUrlFREE Full Text
  8. ↵
    1. Figueroa JF,
    2. Wang DE,
    3. Jha AK
    . Characteristics of hospitals receiving the largest penalties by US pay-for-performance programmes. BMJ Qual Saf. 2016;25(11):898–900
    OpenUrlFREE Full Text
  9. ↵
    1. Bowles KH,
    2. Hanlon A,
    3. Holland D,
    4. Potashnik SL,
    5. Topaz M
    . Impact of discharge planning decision support on time to readmission among older adult medical patients. Prof Case Manag. 2014;19(1):29–38
    OpenUrl
    1. Greenwald JL,
    2. Jack BW
    . Preventing the preventable: reducing rehospitalizations through coordinated, patient-centered discharge processes. Prof Case Manag. 2009;14(3):135–140; quiz 141–142
    OpenUrlCrossRefPubMed
  11. ↵
    1. Wu S,
    2. Tyler A,
    3. Logsdon T,
    4. et al
    . A quality improvement collaborative to improve the discharge process for hospitalized children. Pediatrics. 2016;138(2):e20143604
    OpenUrlAbstract/FREE Full Text
  12. ↵
    1. Zhou H,
    2. Della PR,
    3. Roberts P,
    4. Goh L,
    5. Dhaliwal SS
    . Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review. BMJ Open. 2016;6(6):e011060
    OpenUrlAbstract/FREE Full Text
  13. ↵
    1. Kansagara D,
    2. Englander H,
    3. Salanitro A,
    4. et al
    . Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306(15):1688–1698
    OpenUrlCrossRefPubMed
  14. ↵
    1. Leary JC,
    2. Price LL,
    3. Scott CER,
    4. Kent D,
    5. Wong JB,
    6. Freund KM
    . Developing prediction models for 30-day unplanned readmission among children with medical complexity. Hosp Pediatr. 2019;9(3):201–208
    OpenUrlAbstract/FREE Full Text
  15. ↵
    3M Health Information Systems. Potentially preventable readmissions classification system. 2015. Available at: http://multimedia.3m.com/mws/media/849903O/3m-ppr-grouping-software-fact-sheet.pdf. Accessed July 18, 2019
  16. ↵
    1. Lagoe RJ,
    2. Nanno DS,
    3. Luziani ME
    . Quantitative tools for addressing hospital readmissions. BMC Res Notes. 2012;5:620
    OpenUrlPubMed
  17. ↵
    1. Jiang R,
    2. Wolf S,
    3. Alkazemi MH,
    4. et al
    . The evaluation of three comorbidity indices in predicting postoperative complications and readmissions in pediatric urology. J Pediatr Urol. 2018;14(3):244.e1–244.e7
    OpenUrl
  18. ↵
    1. Medford-Davis LN,
    2. Shah R,
    3. Kennedy D,
    4. Becker E
    . Factors associated with potentially preventable pediatric admissions vary by diagnosis: findings from a large state. Hosp Pediatr. 2016;6(10):595–606
    OpenUrlAbstract/FREE Full Text
  19. ↵
    1. Charlson ME,
    2. Pompei P,
    3. Ales KL,
    4. MacKenzie CR
    . A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–383
    OpenUrlCrossRefPubMed
  20. ↵
    1. Ehwerhemuepha L,
    2. Finn S,
    3. Rothman M,
    4. Rakovski C,
    5. Feaster W
    . A novel model for enhanced prediction and understanding of unplanned 30-day pediatric readmission. Hosp Pediatr. 2018;8(9):578–587
    OpenUrlAbstract/FREE Full Text
  21. ↵
    1. Graham KL,
    2. Auerbach AD,
    3. Schnipper JL,
    4. et al
    . Preventability of early versus late hospital readmissions in a national cohort of general medicine patients. Ann Intern Med. 2018;168(11):766–774
    OpenUrl
  22. ↵
    1. Graham KL,
    2. Dike O,
    3. Doctoroff L,
    4. et al
    . Preventability of early vs. late readmissions in an academic medical center. PLoS One. 2017;12(6):e0178718
    OpenUrl
  23. ↵
    1. Gay JC,
    2. Agrawal R,
    3. Auger KA,
    4. et al
    . Rates and impact of potentially preventable readmissions at children’s hospitals. J Pediatr. 2015;166(3):613–619.e5
    OpenUrlCrossRefPubMed
  24. ↵
    1. Auger KA,
    2. Mueller EL,
    3. Weinberg SH,
    4. et al
    . A validated method for identifying unplanned pediatric readmission. J Pediatr. 2016;170:105–12.e1–2
    OpenUrlPubMed
  25. ↵
    1. Berry JG,
    2. Toomey SL,
    3. Zaslavsky AM,
    4. et al
    . Pediatric readmission prevalence and variability across hospitals [published correction appears in JAMA. 2013;309(10):986]. JAMA. 2013;309(4):372–380
    OpenUrlCrossRefPubMed
  26. ↵
  27. ↵
    1. Pasquali SK,
    2. He X,
    3. Jacobs JP,
    4. et al
    . Measuring hospital performance in congenital heart surgery: administrative versus clinical registry data. Ann Thorac Surg. 2015;99(3):932–938
    OpenUrlCrossRefPubMed
  28. ↵
    1. Feudtner C,
    2. Levin JE,
    3. Srivastava R,
    4. et al
    . How well can hospital readmission be predicted in a cohort of hospitalized children? A retrospective, multicenter study. Pediatrics. 2009;123(1):286–293
    OpenUrlAbstract/FREE Full Text
  29. ↵
    1. Neuman MI,
    2. Alpern ER,
    3. Hall M,
    4. et al
    . Characteristics of recurrent utilization in pediatric emergency departments. Pediatrics. 2014;134(4). Available at: www.pediatrics.org/cgi/content/full/134/4/e1025
  30. ↵
    1. Coller RJ,
    2. Nelson BB,
    3. Sklansky DJ,
    4. et al
    . Preventing hospitalizations in children with medical complexity: a systematic review. Pediatrics. 2014;134(6). Available at: www.pediatrics.org/cgi/content/full/134/6/e1628
  31. ↵
    1. Williams K,
    2. Thomson D,
    3. Seto I,
    4. et al
    ; StaR Child Health Group. Standard 6: age groups for pediatric trials. Pediatrics. 2012;129(suppl 3):S153–S160
    OpenUrlFREE Full Text
  32. ↵
    1. Hardin AP,
    2. Hackell JM
    ; Committee on Practice and Ambulatory Medicine. Age limit of pediatrics. Pediatrics. 2017;140(3):e20172151
    OpenUrlAbstract/FREE Full Text
  33. ↵
    1. Auger KA,
    2. Simon TD,
    3. Cooperberg D,
    4. et al
    . Summary of STARNet: seamless transitions and (Re)admissions network. Pediatrics. 2015;135(1):164–175
    OpenUrlAbstract/FREE Full Text
  34. ↵
    1. Ouellette JA,
    2. Wood W
    . Habit and intention in everyday life: the multiple processes by which past behavior predicts future behavior. Psychol Bull. 1998;124(1):54–74
    OpenUrlCrossRef
  35. ↵
    1. Clewley D,
    2. Rhon D,
    3. Flynn T,
    4. Koppenhaver S,
    5. Cook C
    . Health seeking behavior as a predictor of healthcare utilization in a population of patients with spinal pain. PLoS One. 2018;13(8):e0201348
    OpenUrl
  36. ↵
    1. Simon TD,
    2. Cawthon ML,
    3. Stanford S,
    4. et al
    ; Center of Excellence on Quality of Care Measures for Children with Complex Needs (COE4CCN) Medical Complexity Working Group. Pediatric medical complexity algorithm: a new method to stratify children by medical complexity. Pediatrics. 2014;133(6). Available at: www.pediatrics.org/cgi/content/full/133/6/e1647
  37. ↵
    1. Simon TD,
    2. Cawthon ML,
    3. Popalisky J,
    4. Mangione-Smith R
    ; Center of Excellence on Quality of Care Measures for Children with Complex Needs (COE4CCN). Development and validation of the pediatric medical complexity algorithm (PMCA) version 2.0. Hosp Pediatr. 2017;7(7):373–377
    OpenUrlAbstract/FREE Full Text
  38. ↵
    1. Feudtner C,
    2. Feinstein JA,
    3. Zhong W,
    4. Hall M,
    5. Dai D
    . Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantation. BMC Pediatr. 2014;14:199
    OpenUrlCrossRefPubMed
    1. Hain PD,
    2. Gay JC,
    3. Berutti TW,
    4. Whitney GM,
    5. Wang W,
    6. Saville BR
    . Preventability of early readmissions at a children’s hospital. Pediatrics. 2013;131(1). Available at: www.pediatrics.org/cgi/content/full/131/1/e171
    1. Bardach NS,
    2. Vittinghoff E,
    3. Asteria-Peñaloza R,
    4. et al
    . Measuring hospital quality using pediatric readmission and revisit rates. Pediatrics. 2013;132(3):429–436
    OpenUrlAbstract/FREE Full Text
    1. Berry JG,
    2. Hall M,
    3. Cohen E,
    4. O’Neill M,
    5. Feudtner C
    . Ways to identify children with medical complexity and the importance of why. J Pediatr. 2015;167(2):229–237
    OpenUrlCrossRefPubMed
  42. ↵
    Agency for Healthcare Research and Quality. HCUP chronic condition indicator. 2016. Available at: https://www.hcup-us.ahrq.gov/toolssoftware/chronic_icd10/chronic_icd10.jsp. Accessed July 18, 2019
  43. ↵
    1. Chen T,
    2. Guestrin C
    , XGBoost: a scalable tree boosting system. In: KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; August 13-17, 2016; San Francisco, CA. 785–794.
  44. ↵
    1. Morgan DJ,
    2. Bame B,
    3. Zimand P,
    4. et al
    . Assessment of machine learning vs standard prediction rules for predicting hospital readmissions. JAMA Netw Open. 2019;2(3):e190348
    OpenUrl
  45. ↵
    1. Lan Z,
    2. Zhou G,
    3. Duan Y,
    4. Yan W
    . AI-Assisted Prediction on Potential Health Risks with Regular Physical Examination Records. In: 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC); June 18-21, 2018 Guangzhou, China. 346–352
    1. Allyn J,
    2. Allou N,
    3. Augustin P,
    4. et al
    . A comparison of a machine learning model with EuroSCORE II in predicting mortality after elective cardiac surgery: a decision curve analysis. PLoS One. 2017;12(1):e0169772
    OpenUrl
  47. ↵
    1. Casey P,
    2. Zolfaghar K,
    3. Eckert C,
    4. et al
    . 12 predicting patients at risk for leaving without being seen using machine learning. Ann Emerg Med. 2018;72(4):S5–S6
    OpenUrl
  48. ↵
    1. Snijders TAB,
    2. Bosker R
    . Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. 1st ed. Thousand Oaks, CA: Sage Publications Ltd; 1999
  49. ↵
    1. Alba AC,
    2. Agoritsas T,
    3. Walsh M,
    4. et al
    . Discrimination and calibration of clinical prediction models: users’ guides to the medical literature. JAMA. 2017;318(14):1377–1384
    OpenUrlCrossRefPubMed
  50. ↵
    1. Schmid CH,
    2. Griffith JL
    . Multivariate classification rules: calibration and discrimination. Available at: https://doi.org/10.1002/9781118445112.stat05650. Accessed January 17, 2020
  51. ↵
    1. DeLong ER,
    2. DeLong DM,
    3. Clarke-Pearson DL
    . Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–845
    OpenUrlCrossRefPubMed
  52. ↵
    1. Crowson CS,
    2. Atkinson EJ,
    3. Therneau TM
    . Assessing calibration of prognostic risk scores. Stat Methods Med Res. 2016;25(4):1692–1706
    OpenUrlCrossRefPubMed
  53. ↵
    1. Muller CJ,
    2. MacLehose RF
    . Estimating predicted probabilities from logistic regression: different methods correspond to different target populations. Int J Epidemiol. 2014;43(3):962–970
    OpenUrlCrossRefPubMed
  54. ↵
    1. Ribeiro MT,
    2. Singh S,
    3. Guestrin C
    . “Why should I trust you?” Explaining the predictions of any classifier. 2016. Available at: http://arxiv.org/abs/1602.04938. Accessed August 12, 2018
  55. ↵
    1. Rice H,
    2. Say R,
    3. Betihavas V
    . The effect of nurse-led education on hospitalisation, readmission, quality of life and cost in adults with heart failure. A systematic review. Patient Educ Couns. 2018;101(3):363–374
    OpenUrl
    1. Brown EG,
    2. Burgess D,
    3. Li CS,
    4. Canter RJ,
    5. Bold RJ
    . Hospital readmissions: necessary evil or preventable target for quality improvement. Ann Surg. 2014;260(4):583–589; discussion 589–591
    OpenUrlCrossRefPubMed
  57. ↵
    1. Whitaker-Brown CD,
    2. Woods SJ,
    3. Cornelius JB,
    4. Southard E,
    5. Gulati SK
    . Improving quality of life and decreasing readmissions in heart failure patients in a multidisciplinary transition-to-care clinic. Heart Lung. 2017;46(2):79–84
    OpenUrl
  58. ↵
    1. Auger KA,
    2. Kenyon CC,
    3. Feudtner C,
    4. Davis MM
    . Pediatric hospital discharge interventions to reduce subsequent utilization: a systematic review. J Hosp Med. 2014;9(4):251–260
    OpenUrlCrossRefPubMed
  59. ↵
    1. Curran JA,
    2. Taylor A,
    3. Chorney J,
    4. et al
    . Development and feasibility testing of the pediatric emergency discharge interaction coding scheme. Health Expect. 2017;20(4):734–741
    OpenUrl
    1. Halasyamani L,
    2. Kripalani S,
    3. Coleman E,
    4. et al
    . Transition of care for hospitalized elderly patients--development of a discharge checklist for hospitalists. J Hosp Med. 2006;1(6):354–360
    OpenUrlCrossRefPubMed
  61. ↵
    1. Smith MA,
    2. Clayman ML,
    3. Frader J,
    4. et al
    . A descriptive study of decision-making conversations during pediatric intensive care unit family conferences. J Palliat Med. 2018;21(9):1290–1299
    OpenUrl
  62. ↵
    1. Gonçalves-Bradley DC,
    2. Lannin NA,
    3. Clemson LM,
    4. Cameron ID,
    5. Shepperd S
    . Discharge planning from hospital. Cochrane Database Syst Rev. 2016;(1):CD000313
  63. ↵
    1. Kang JH,
    2. Kim CW,
    3. Lee SY
    . Nurse-perceived patient adverse events depend on nursing workload. Osong Public Health Res Perspect. 2016;7(1):56–62
    OpenUrl
    1. Dye E,
    2. Wells N
    . Subjective and objective measurement of neonatal nurse practitioner workload. Adv Neonatal Care. 2017;17(4):E3–E12
    OpenUrl
  65. ↵
    1. Lake ET,
    2. de Cordova PB,
    3. Barton S,
    4. et al
    . Missed nursing care in pediatrics. Hosp Pediatr. 2017;7(7):378–384
    OpenUrlAbstract/FREE Full Text
  66. ↵
    1. Eriksson CO,
    2. Stoner RC,
    3. Eden KB,
    4. Newgard CD,
    5. Guise JM
    . The association between hospital capacity strain and inpatient outcomes in highly developed countries: a systematic review. J Gen Intern Med. 2017;32(6):686–696
    OpenUrl
  67. ↵
    1. Kohn R,
    2. Harhay MO,
    3. Bayes B,
    4. et al
    . Ward capacity strain: a novel predictor of 30-day hospital readmissions. J Gen Intern Med. 2018;33(11):1851–1853
    OpenUrl
  68. ↵
    1. Hesselink G,
    2. Zegers M,
    3. Vernooij-Dassen M,
    4. et al
    ; European HANDOVER Research Collaborative. Improving patient discharge and reducing hospital readmissions by using Intervention Mapping. BMC Health Serv Res. 2014;14(1):389
    OpenUrlCrossRefPubMed
  69. ↵
    1. Cavanaugh JJ,
    2. Jones CD,
    3. Embree G,
    4. et al
    . Implementation Science Workshop: primary care-based multidisciplinary readmission prevention program. J Gen Intern Med. 2014;29(5):798–804
    OpenUrlCrossRefPubMed
Back to top

In this issue

View this article with LENS
Email Article
Request Permissions
Article Alerts
Citation Tools
Development and Validation of a Web-Based Pediatric Readmission Risk Assessment Tool
Thom Taylor, Danielle Altares Sarik, Daria Salyakina
Hospital Pediatrics Mar 2020, 10 (3) 246-256; DOI: 10.1542/hpeds.2019-0241
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
Print
Download PDF
Insight Alerts

Jump to section

Subjects