OBJECTIVES: Diagnoses extracted from physician notes are used to calculate hospital quality metrics; failure to document high-risk diagnoses may lead to the appearance of worse-than-expected outcomes for complex patients. Academic hospitals often rely on documentation authored by trainees, yet residents receive little training in this regard. In this study, we evaluate inpatient pediatric resident notes to determine which high-risk diagnoses are commonly missed and assess the efficacy of a multitiered intervention to improve the documentation of these diagnoses.
METHODS: In a baseline review of 220 charts, 13 frequently missed high-risk diagnoses were identified in 2013. Interventions began in 2014, including physician education and reference cards. The intervention also included note template prompts for 4 of the diagnoses. Using a standardized rubric, we reviewed charts for 3 years (2013, 2014, and 2015). The average within-disease probability of missed high-risk diagnoses was compared across time.
RESULTS: There was a decrease in the probability of undocumented target high-risk diagnoses after the intervention (52% vs 36% in 2014 [odds ratio = 0.51; P < .001] and 37% in 2015 [odds ratio = 0.50; P < .001]). Documentation of diagnoses prompted by the note template was not significantly better than those targeted by the other interventions alone (P = .55).
CONCLUSIONS: Pediatric residents were significantly less likely to omit a high-risk diagnosis in their notes after implementation of our documentation improvement program, suggesting that curriculum development is an effective method of improving documentation, with the goal of improving the accuracy of health systems performance indices.
As pediatric hospital staffs work toward greater quality and patient safety, accurate outcome data are vital. Many children’s hospital staffs now submit these data to external sources as part of quality of care reporting and are moving toward increased transparency for patients and consumers, making outcome data for a given institution, department, or even individual provider increasingly available to the public.1–4 Importantly, the information used to calculate quality metrics, such as the mortality index, is collected from physician documentation by certified medical coders as they capture diagnoses for billing. If the coded diagnoses fail to illustrate the severity of a patient’s illness because of incomplete physician documentation, aggregated quality data will not appropriately reflect the patient population’s complexity, leading to an appearance of inappropriately high mortality rates.2,5 On the basis of established mathematical models, each International Classification of Diseases diagnosis is weighted according to the magnitude of its effect on hospital resource use and on an individual patient’s severity of illness and risk of mortality.1,2,6,7 If a patient has a greater number of high-risk diagnoses, such as sepsis, pancytopenia, and thousands of others, that patient will be assigned a higher expected mortality risk. However, coders are not medically trained and cannot infer diagnoses that have not been explicitly documented by providers. For example, a physician may write that a patient is febrile, tachycardic, and has a positive blood culture result, but if the term “sepsis” is not stated, it cannot be coded and will not be reflected in the patient’s estimated mortality risk.
Clinical documentation improvement (CDI) programs, which are designed to ensure accurate documentation of these high-risk diagnoses, are now common in internal medicine and surgical fields, largely driven by clear financial incentives under Medicare payment structures.8–14 However, these financial incentives may not translate to pediatrics given the regional pay structure differences among children’s hospitals, so many pediatricians may not be supported by formal CDI initiatives.15 Additionally, residents often receive no formal training on CDI concepts despite being responsible for most documentation at academic medical centers.16–18
To date, there are no published studies on pediatric CDI efforts. Our goal with this study was to establish a clinical documentation curriculum at our institution and to examine its efficacy in improving documentation of high-risk diagnoses, with a greater aim of improving the accuracy of our quality data.
We performed this single-center time series study at a tertiary care academic children’s hospital within a hospital. The study was determined to be exempt from human subject research review by the Institutional Review Board at the University of California, Los Angeles.
To assess current documentation practices, we reviewed 200 charts that were randomly selected from all 1119 pediatric ward and ICU discharges between July 1 and December 31, 2013. We also reviewed all 20 mortalities from the calendar year because these patients were presumed more likely to have high-risk diagnoses. This resulted in a total of 220 baseline charts reviewed. Eighty-four (38%) of these 220 charts were ICU patients as defined by spending time in the ICU at any point during that encounter. Four pediatric hospitalist attending physicians were trained in reviewing charts, specifically looking for missed high-risk diagnoses as defined by the Centers for Medicare and Medicaid Services.19 Only notes written by residents were reviewed. Of the baseline charts reviewed, 67% had 1 or more missing high-risk diagnosis. Diagnoses that were missed 5 or more times were targeted for intervention, resulting in the following list of 13 diagnoses: acidosis, alkalosis, acute kidney injury, chronic renal failure, epilepsy, heart failure, hypertension, malnutrition, neutropenia, anemia, pancytopenia, sepsis, and shock.
Interventions began in July 2014 at the start of the new academic year. Educational conferences on the importance of documenting specific diagnoses in terms that coders will recognize were presented to faculty and residents. These lectures were repeated over the course of several months in resident orientation, noon conferences, departmental quality meetings, and faculty meetings. A reference card (Fig 1) defining the target diagnoses was distributed electronically to residents and posted in their workrooms to educate them about when these diagnoses apply. The diagnoses were defined by using established literature; when no clear guidelines were available, we relied on expert consensus among subspecialists at our institution. We also revised the electronic medical record (EMR) note templates, which are used by all our pediatric residents, to include 4 of the target diagnoses that were easiest to incorporate without cluttering the note or overly encumbering note writers. Drop-down lists that were already present within the templates were modified to specifically mention the diagnoses of acute kidney injury, anemia, pancytopenia, and sepsis to prompt residents to include them when applicable.
A standardized rubric was created to specifically look for target diagnoses (Supplemental Fig 3). Using this rubric, the 4 reviewers achieved >90% interrater agreement on a test sample of 10 charts before initiation of data collection. A total of 100 charts were then randomly selected from ward and ICU discharges between July 1, 2013, and December 31, 2013, for our preintervention data and 2014 and 2015 for postintervention data. The 100 charts selected from 2013 were unique for this phase of the study because we were now reviewing charts that were focused only on the 13 diagnoses targeted after the baseline review. This sample size was selected on the basis of an a priori power analysis (80% power, 5% α) to detect a 20% reduction in missed diagnoses. Additionally, all mortalities during those intervals were reviewed. Newborn nursery, NICU, and cardiothoracic ICU charts were excluded because these populations have a unique set of diagnoses that are less generalizable to other patients. Patients admitted for 2 or fewer calendar days were also excluded because they were less likely to have high-risk diagnoses. Charts that were found to meet exclusion criteria after chart review began were omitted without being replaced. This resulted in a total of 97 reviewed charts in 2013, 117 in 2014, and 114 in 2015 after applying inclusion and exclusion criteria. Each target diagnosis was coded as “present and documented,” “present but not documented,” or “not present” within each chart.
The primary outcome of this study was a reduction in undocumented target high-risk diagnoses after the intervention. Secondary outcomes were the efficacy of EMR templates and whether improved documentation would lead to an increase in the expected mortality (ie, the percent of patients expected to die on the basis of illness severity) as measured by Vizient, formerly known as the University Health-System Consortium.20 The average within-disease probability of missed high-risk diagnoses was compared across time (preintervention and postintervention) by using a mixed effects logistic regression model with disease random intercept. An interaction P value for the differences between years was used to assess the utility of the note template. Differences in expected mortality were compared by using an independent samples test of proportions.
Overall in 2013, there was a 52% probability of a target diagnosis being “present but not documented” (95% confidence interval [CI]: 0.47 to 0.57), which decreased to 36% in 2014 (95% CI: 0.32 to 0.40) after implementation of the interventions (odds ratio = 0.51; P < .001; Table 1). This reduction was sustained in 2015 with a 37% probability (P < .001; 95% CI: 0.32 to 0.42). As for the individual diagnoses, there was a reduction in missed documentation for 3 of the 13 target diagnoses (Fig 2). The results for the remainder of the diagnoses are available in Supplemental Table 2.
The percent of the total charts reviewed per year that were mortalities remained consistent over the study interval (13%, 15%, and 16% in 2013, 2014, and 2015, respectively; P = .89). The percent of patients with an ICU stay also remained the same (42%, 33%, and 39% in 2013, 2014, and 2015, respectively; P = .32). When further analyzed by ICU status, high-risk diagnoses were more prevalent in ICU patients compared with ward patients (26% vs 15.7%; P < .001). There was sustained improvement in missed diagnoses in ward charts, which went from 59% in 2013 to 38% in 2014 (P < .001) and 39% in 2015 (P < .001), but the improvement in ICU charts was not statistically significant.
For our secondary outcome measures, documentation of diagnoses included in the note template was not significantly better than those targeted by educational interventions alone (P = .55). Expected mortality rates did increase by 22% over the study interval, but this was not statistically significant.
Our study revealed that before our interventions, the probability of missing a target diagnosis was >50%, suggesting that our patient population had a higher risk of mortality than our coded data would indicate. The educational intervention was effective at improving inpatient pediatric resident documentation of high-risk diagnoses and sustaining improvement with ongoing education. This approach may be particularly useful for pediatric institutions without funding for formal CDI programs that may need to rely on physician-driven initiatives.
As expected, ICU patients had more high-risk diagnoses than ward patients, but the 2 groups were similarly susceptible to omissions in documentation; in fact, only the ward patient group showed statistically significant improvement. Thus, on the basis of our data, CDI efforts should include lower-acuity patients as well as ICU patients. We had anticipated that an increased ability to capture high-risk diagnoses would lead to an increased expected mortality, but this was not statistically significant.
Similar measures such as lecture series and reference cards have been used in publications in adult literature to achieve a 43% increase in expected mortality and increases in reimbursement up to 16% to 24%.21,22 However, many of these institutions also retain the services of a CDI team that audits notes in real time and queries physicians regarding missed diagnoses that may need to be added to their documentation.11 In fact, at the time of this study, our institution’s CDI specialists were reviewing all adult inpatient documentation but not pediatric charts because of funding limitations and the lack of perceived benefits. As a result of our work, our institution will be providing resources for a CDI team that will use our reference card to query documentation errors in real time, further bridging the gap between what physicians document and what coders interpret.
A notable strength of our study was that we evaluated undocumented and documented diagnoses as a percent of the total number of instances in which the diagnosis was clinically applicable. In this way, we were able to control for changes in patient complexity over time, in contrast to previous studies in which the authors only looked at absolute numbers of documented high-risk diagnoses, which may be confounded by increases in the prevalence of studied diagnoses over time.
A limitation of our study was its single-center design that was focused solely on resident documentation, so our data may not be generalizable to nonteaching institutions. Additionally, because our educational interventions consisted of multiple elements, we cannot comment on the effectiveness of any single component. The retrospective nature of the review made it impossible to determine the “false-positive” rate, that is, the diagnoses that were documented but may not in fact have been applicable. The phenomenon of manipulating documentation to maximize reimbursement has been labeled “diagnosis-related group creep” and is a potential adverse consequence of CDI efforts.23 Finally, although we as physician reviewers had strong interrater agreement with each other, we were unable to correlate our findings with our coders because they review all physician notes, whereas in our study, we only reviewed resident documentation.
Our educational interventions successfully decreased the probability of missing diagnoses by 16%; however, there are still significant opportunities for further improvements to ensure that our quality metrics reflect the complexity of our patient population. Future work will expand on the current intervention to target additional diagnoses and to include attending and fellow documentation. There also may be more applications of EMR technology to assist in documentation on the basis of existing data in other areas of the chart, such as importing physician-generated “problem lists” into daily notes to make it accessible to coders. In further studies, authors may also assess the effect of these interventions on resident experience. Encouraging use of more precise language may help trainees recognize and treat high-risk diagnoses more promptly, but it is important to identify and prevent any adverse effects on resident workflow.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Although no direct funding was required, statistical support was received through the National Institutes of Health National Center for Advancing Translational Science University of California Los Angeles Clinical and Translational Science Institute grant UL1TR001881.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- ↵University Health System Consortium. Mortality risk adjustment methodology for University Health System’s clinical data base. 2008. Available at: https://archive.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/mortality/Meurer.pdf. Accessed November 1, 2016
- U.S. Centers for Medicare and Medicaid Services. Hospital compare. Available at: https://www.medicare.gov/hospitalcompare/search.html. Accessed November 26, 2017
- ↵The Joint Commission. Quality check. Available at: http://www.healthcarequalitydata.org/. Accessed November 26, 2017
- Frazee RC,
- Matejicka AV II.,
- Abernathy SW,
- et al
- Kittinger BJ,
- Matejicka A II.,
- Mahabir RC
- Momin SR,
- Lorenz RR,
- Lamarre ED
- Reyes C,
- Greenbaum A,
- Porto C,
- Russell JC
- Yount KW,
- Reames BN,
- Kensinger CD,
- et al
- ↵Centers for Medicare and Medicaid Services. Draft ICD-10-CM/PCS MS-DRGv31.0 definitions manual. Available at: https://www.cms.gov/icd10manual/version31-fullcode-cms/P0001.html. Accessed November 16, 2016
- ↵VIZIENT Clinical Data Base/Resource Manager. Patient outcomes by hospital/hospital system. Available at: https://www.vizientinc.com/our-networks/networks-for-academic-medical-centers. Accessed September 13, 2016
- Spellberg B,
- Harrington D,
- Black S,
- Sue D,
- Stringer W,
- Witt M
- Zalatimo O,
- Ranasinghe M,
- Harbaugh RE,
- Iantosca M
- Copyright © 2018 by the American Academy of Pediatrics