Administrative Data Misclassifies and Fails to Identify Nephrotoxin-Associated Acute Kidney Injury in Hospitalized Children
Objectives: Nephrotoxin exposure is a common cause of acute kidney injury (AKI) in hospitalized children. AKI detection relies on regular serum creatinine (SCr) screening among exposed patients. We sought to determine how well administrative data identify hospitalized noncritically ill children with nephrotoxic medication-associated AKI in the contexts of incomplete and complete screening.
Methods: We conducted a single-center retrospective cohort study among noncritically ill hospitalized children. We compared administrative data sensitivity to that among a separate cohort for whom adequate screening was defined as daily SCr measurement. For the original cohort, nephrotoxin exposure was defined as exposure to ≥3 nephrotoxins at once or ≥3 days of aminoglycoside therapy. AKI was defined by the change in SCr (pediatric-modified Risk Injury Failure Loss End-Stage Renal Disease [pRIFLE] criteria) or discharge code. Adequate SCr screening was defined as 2 measurements obtained ≤96 hours apart. Administrative data and laboratory values were merged to compare AKI by discharge code and pRIFLE criteria.
Results: 747 of 1472 (50.7%) nephrotoxin-exposed patients were adequately screened; 82 (11.0%) had AKI by pRIFLE criteria, 52 (7.0%) by discharge code. Sensitivity of nephrotoxin-associated AKI diagnosis by discharge code compared with pRIFLE criteria was 23.2% (95% confidence interval = 14.0–32.3). In the comparison cohort, 70 (26.8%) patients had AKI by pRIFLE criteria and 26 (10.0%) by discharge code; sensitivity was 21.4% (95% confidence interval = 11.8%–31.0%).
Conclusions: pRIFLE criteria identified more patients than were identified by discharge code. Identifying patients with nephrotoxin-associated AKI by discharge code, even in the presence of complete AKI detection, underrepresents the true incidence of nephrotoxin-associated AKI in hospitalized children.
Acute kidney injury (AKI) is a potentially devastating event that places patients at increased risk of poor outcomes and is strongly linked to chronic kidney disease (CKD) in adults and children.1–4 The incidence of AKI is high and continues to increase,2,5–7 which may lead to increasing prevalence of CKD, which in turn could create long-term burdens on patients and health care systems.8
In hospitalized children, recent studies have demonstrated that the cause of AKI is not primary renal disease, as was previously thought, but the result of a systemic illness or its treatment.9 In noncritically ill hospitalized children, exposure to nephrotoxic medications is among the most common causes of AKI and is associated with significant morbidity, increased length of stay, and cost.10–12 Recent studies have found AKI among noncritically ill patients to be associated with exposure to multiple nephrotoxins and to aminoglycoside antibiotics specifically.11,12 Accurate detection of AKI relies on serial measurements of serum creatinine (SCr) level or urine output over time.13 However, noncritically ill pediatric patients are often not monitored for AKI and providers may miss an opportunity to prevent or diagnose AKI.11,12 The epidemiology of nephrotoxin-associated AKI on a large scale is therefore not well understood.
Administrative databases are a potentially useful means to better define AKI epidemiology on a broad scale because of the large number of patients included.14 Although administrative data capture some diagnoses well,15–18 for other diagnoses, administrative data are inherently inaccurate relative to clinical diagnoses.19–23 Studying nephrotoxin-associated AKI in noncritically ill hospitalized children using administrative data has not been attempted. This is likely because of the absence of laboratory or patient urine output data, which are key to accurate diagnosis of AKI. The purpose of this study was to assess the sensitivity of administrative data to detect nephrotoxin-associated AKI among noncritically ill hospitalized patients. We hypothesized that, given infrequent monitoring of SCr, administrative data would not detect AKI effectively. We also hypothesized that if applied to a cohort in which monitoring was complete, the detection sensitivity would increase.
We conducted a single-center retrospective study in which administrative data were merged with laboratory values to allow direct comparison of AKI by discharge code and pediatric-modified Risk, Injury, Failure, Loss, and End-Stage Renal Disease (pRIFLE) values, respectively.
SCr data were obtained from our hospital electronic health record. Data including demographics, all-patient refined diagnosis-related group (APR-DRG)–defined service line,24 medications, and discharge coding for this study were obtained from the Pediatric Health Information System (PHIS). PHIS is an administrative database that contains inpatient, emergency department, ambulatory surgery, and observation data from 43 not-for-profit, tertiary care pediatric hospitals in the United States. These hospitals are affiliated with the Children’s Hospital Association (Overland Park, KS). Data quality and reliability are ensured through a joint effort between the Children’s Hospital Association and participating hospitals. The data warehouse function for the PHIS database is managed by Truven Health Analytics (Ann Arbor, MI). For the purposes of external benchmarking, participating hospitals provide discharge/encounter data including demographics, diagnoses, and procedures. Forty-two of these hospitals also submit resource utilization data (eg, pharmaceuticals, imaging, and laboratory) into PHIS. Data are deidentified at the time of data submission and subjected to a number of reliability and validity checks before being included in the database. PHIS allows a member hospital to reidentify patients so that medical record data can be linked to administrative data.
We have previously described a cohort of nephrotoxin-exposed patients who were screened daily by SCr and tracked prospectively for development of AKI as part of a local quality improvement project. Details of the method of collection and database composition, compilation of the nephrotoxin list, and nephrotoxin exposure criteria are the subject of a separate report, which does not assess International Classification of Diseases, Ninth Revision (ICD-9) codes.25 As part of this quality improvement project, ∼99% of nephrotoxin-exposed patients had daily SCr measurement during their exposure. For the current study, patients with an admission and discharge date from June 1, 2011 through June 30, 2012 with a matching PHIS admission were included.
Nephrotoxin exposure was defined as exposure to ≥3 nephrotoxins at once or ≥3 days of intravenous aminoglycoside antibiotic therapy.11 Exposure to a dose of intravenous contrast material was considered 1 nephrotoxin exposure for the day of, and the day after the infusion. Adequate SCr screening was defined as 2 measurements obtained ≤96 hours apart during an admission. This time period was chosen because we knew that daily screening was unlikely11,12 and estimated that at least half of nephrotoxin-exposed patients would be screened every 96 hours (unpublished data). For the comparison cohort, exposure definitions were identical except for intravenous contrast material, for which exposure counted toward the day of and 6 days after the infusion, and adequate screening, which was defined as SCr measurement every day of exposure.25
AKI was defined by ICD-9 code (584, 584.5, 584.6, 584.7, 584.8, 586, 584.9, and 593.9, 788.5) or by any of 3 strata (R, I, and F) of the SCr-based pRIFLE.11,12 AKI as determined by pRIFLE has been associated with poor outcomes in critically ill and noncritically ill children with AKI.12,13,26,27 Because nephrotoxin-associated AKI is usually nonoliguric in nature28 and urine output data were unavailable, we did not use the pRIFLE urine output criteria. The pRIFLE-R stratus is considered injury by its 25% decrease in estimated creatinine clearance and was therefore included in this study.13 The pRIFLE-L and E strata are long-term outcome measures of AKI and were not considered for this study.
All analyses were performed by using SAS Software version 9.3 (SAS Institute Inc, Cary, NC). First, records for all admissions to CCHMC from January 1, 2009 through April 30, 2011, including medications administered, clinical service line, APR-DRG, underlying medical condition, and all discharge diagnoses were downloaded from the PHIS database. The following patients were excluded during the download: (1) those requiring intensive care because the etiology of AKI in these patients differs significantly from noncritically ill patients; (2) those with chronic kidney disease (defined as ICD-9 codes 188.×, 189.×, 209.64, 250.4×, 403, 404, 580.×, 581.×, 582.×, 583, 585.6, V42.0, and E879.1) to restrict our cohort to new-onset AKI; (3) those >18 years old; (4) those with admissions <24 hours in length (short-stay admissions) because SCr measurements from these admissions would be unlikely to identify an acute change from a baseline value.
To identify patients, deidentified PHIS medical record numbers (MRNs) were converted to their original state. Next, the list of MRNs was used to download laboratory data from a separate, local database. The 2 datasets were merged using the combination of MRN, admission date, and discharge date as an identifier of each unique admission and operational definitions described earlier were applied. Descriptive analysis was conducted in 3 groups: (1) the entire cohort, (2) all nephrotoxin-exposed patients, and (3) nephrotoxin-exposed patients screened adequately. Comparisons were made by group and not in pairwise fashion. Because of the increased statistical power provided by our large sample size, the P value for declaring statistical significance in these group comparisons was set at .001 to reduce likelihood of a false-positive result. Change in SCr level was determined for pairs of values obtained for the same patient during the same admission no more than 96 hours apart. SCr values were ordered sequentially by date and each result was iteratively divided by the preceding result. For nephrotoxin-exposed patients screened adequately, the sensitivity, specificity, and positive and negative predictive values of ICD-9 codes for AKI were calculated in a standard 2 × 2 table using pRIFLE as the gold standard.
PHIS data were extracted, merged with existing AKI data from the project database, and subjected to the same exclusion criteria described above. Descriptive analysis was conducted as described earlier. Sensitivity, specificity, and positive and negative predictive values were calculated in a standard 2 × 2 table using pRIFLE as the gold standard.
Human Subject Protection
Our institutional review board reviewed the project and considered it to be excluded from human subjects research. Informed consent beyond the standard consent for treatment of all inpatients was not required.
Among the 38 410 patients discharged from our hospital during the study period, there were 28 753 patients who met inclusion criteria. Of these, 1472 patients were nephrotoxin-exposed (5.1%), 747 (50.7%) of whom had adequate SCr screening (Fig 1).
Demographics of nephrotoxin-exposed versus non–nephrotoxin-exposed patients are depicted in Table 1. Patients with nephrotoxin exposure had significantly longer lengths of stay and were exposed to a higher number of medications than those without nephrotoxin exposure. Nephrotoxin-exposed patients were more likely to carry commercial insurance and unexposed patients more likely to carry Medicaid. Nephrotoxin-exposed patients had a higher proportion of white and Asian patients and lower proportions of American Indian and African American patients than unexposed patients. Exposed patients were significantly more likely to have received care from cancer care and hematology, cardiac care, digestive disease, infectious disease, neonatal care, neuroscience, and solid organ or bone marrow transplant services, whereas unexposed patients were more likely to have received care from other medicine, orthopedics, and other surgery services.
Table 2 depicts the demographics of adequately screened (middle column) versus inadequately screened nephrotoxin-exposed patients (left column) and patients from the comparison cohort (right column). Compared with inadequately screened patients, adequately screened nephrotoxin-exposed patients were younger; exposed to more medications; had a longer length of stay; were more likely to receive services through cancer care and hematology, infectious disease, and organ or bone marrow transplant services; and were more likely to use a form of insurance or payment other than commercial or self-pay. Race was not different between adequately and inadequately screened patients. Of note, all of the exposed organ and bone marrow transplant patients had adequate screening. Adequately screened and comparison cohort patients showed considerable similarity, differing only in length of stay and service line. Comparison cohort patients had longer length of stay, were more likely to receive services through respiratory and organ or bone marrow transplant services and less likely to receive services through cancer care and hematology, infectious disease, and digestive disease services.
Nephrotoxin-Associated AKI Sensitivity
Using pRIFLE criteria, AKI was present in 82 (11.0%) of the 747 nephrotoxin-exposed adequately screened patients. Using discharge code, AKI was present in 52 (7.0%) of the same group. Sensitivity of nephrotoxin-associated AKI diagnosis by discharge code compared with pRIFLE criteria was 23.2% (95% confidence interval [CI] 14.0–32.3), specificity was 95.0% (95% CI 93.1–96.6). The positive predictive value of nephrotoxin-associated AKI by discharge coding was 36.5% (95% CI 23.6–51.0), the negative predictive value 90.9% (95% CI 88.6–93.0). Results of the analysis are depicted in Fig 2A.
Nephrotoxin-Associated AKI Sensitivity, Comparison Cohort
Based on daily SCr screening, pRIFLE criteria identified AKI in 70 (26.8%) of 261 nephrotoxin-exposed patients.25 Using discharge code, AKI was present in 26 (10.0%) of the same group. Sensitivity of nephrotoxin-associated AKI diagnosis by discharge code compared with pRIFLE criteria was 21.4% (95% CI 11.8–31.0), and specificity was 94.2% (95% CI 90.1–97.5). The positive predictive value of nephrotoxin-associated AKI by discharge coding was 57.7% (95% CI 38.7–76.7), and the negative predictive value was 76.6% (95% CI 71.2–82.0). Results of the analysis are depicted in Fig 2B.
The results of this study support the use of SCr-based criteria, such as pRIFLE, rather than reliance on administrative coding data to estimate the incidence of nephrotoxin-associated AKI in the hospitalized noncritically ill pediatric population. However, in the absence of widespread adequate screening by serial SCr measurement or accurate reporting of urine output, pRIFLE cannot be applied reliably and thus may still underestimate nephrotoxin-associated AKI incidence. Our study found 58% more patients with nephrotoxin-associated AKI when applying pRIFLE criteria compared with ICD-9 criteria. The additional cases identified were not due to pRIFLE identifying only less severe cases and ICD-9 codes identifying only the most severe. For the study cohort, when separated by pRIFLE severity class, the sensitivity of detection increased but not considerably as severity of injury increased (Data Not Shown). Thus, pRIFLE criteria appear superior to ICD-9 code no matter how severe the AKI.
The ideal interval for screening for nephrotoxin-associated AKI in exposed patients is unknown. Daily screening, as was conducted in the comparison cohort,25 detected 2.5 times more AKI than screening within 96 hours, as was allowed in the study cohort (28.0% vs 11.0%). Without frequent monitoring of SCr in nephrotoxin-exposed patients, AKI may go undetected. However, the long-term effects of nephrotoxin-associated AKI are currently not well known.9 The justification of frequent, costly SCr measurements is therefore difficult. Given the potential damage that is known from studies of AKI in adult and pediatric patients who did not examine nephrotoxin exposure specifically,4,29,30 we believe more frequent measurement is indicated.
Finding that nephrotoxin-exposed patients had longer lengths of stay and total number of medications administered compared with non-nephrotoxin-exposed patients was not surprising. It is likely that with increased time spent admitted to the hospital, more medications will be administered, and the likelihood of nephrotoxin exposure will increase. This also may be confounding by indication, where sicker patients a priori receive more medications and stay longer. Both of these concepts are consistent with our finding of nephrotoxin-exposed patients predominantly associated with APR-DRG service lines where one would expect to find patients with malignancy, bone marrow transplant, multivisceral and liver transplant, and cystic fibrosis. We have observed these patient populations to be overrepresented among nephrotoxin-exposed patients, likely due to prolonged hospitalization and frequent use of nephrotoxins for treatment.25 Likewise, it is easy to imagine cardiac care (surgical, transplant) and neonatal (premature infants on multiple antibiotics) services among this cohort of patients. It is less clear why nephrotoxin-exposed patients had different insurance coverage and race. This is less likely to indicate an association of socioeconomic status or race with nephrotoxin exposure and more likely to be a result of the specific patient population cared for at our institution. Many patients receiving care for the most complex conditions come from outside the immediate hospital catchment area including a portion from outside the United States. Thus, these patients may skew the racial and insurance status distributions for the exposed population. The predominance of transplant and respiratory service patients in the comparison cohort may have increased the risk of AKI compared with the adequately screened cohort. However, fewer patients associated with an infectious disease, digestive disease, and cancer APR-DRG service line may have balanced some of the risk between the groups. Thus, we felt the nephrotoxin-exposed adequately screened and comparison cohorts could be compared directly.
Administrative data have many advantages for epidemiologic study in hospitalized pediatric populations, including generating large multicenter cohorts of patients to examine exposures and outcomes over time.14 However, administrative data are also subject to limitations, such as an inherent inaccuracy of ICD-9 codes relative to clinical diagnoses.19–23 Administrative data has been used to identify CKD and AKI in adult populations. For CKD, sensitivities showed a high degree of variation, with a median of 41%. AKI sensitivities were considered poor, with a median of 21%.31 The ICD-9 codes used in our study did not match any of the sets used in the aforementioned studies, which were all performed in adults, making direct comparison difficult. Nonetheless, our results are consistent with previous analyses. The disagreement between ICD-9 codes and clinical diagnoses is thought to be multifactorial, including the former being generated by professional billing staff, who are limited in the codes they can use by the specific words used in the clinical notes.23 Given these limitations, we were not surprised to find that use of ICD-9 codes alone identified only 19 of the 82 (23.2%) patients with nephrotoxin-associated AKI by pRIFLE criteria (Fig 2A).
For the comparison cohort, we assumed our near complete case identification and ensured documentation of AKI in the medical record would yield more accurate billing code data. The similarly low sensitivity in our comparison cohort (Fig 2B) was therefore unexpected. This is of particular significance given the reliance on administrative data for public reporting, national benchmark and performance data, and Medicare reimbursement decisions.32–34 Future study is needed to fully understand the apparent disconnect between chart documentation and administrative data to inform population-based estimates of burden and cost of nephrotoxin-associated AKI in the pediatric population.
We thank Anthony Goudie and Matthew Fenchel for assistance in study design and initial analysis and Colleen Mangeot for assistance in data management and analysis.
Preliminary data were presented at the 2013 Pediatric Academic Societies Meeting; May 3–7, 2013; Washington, DC; and at the 2013 Pediatric Hospital Medicine Meeting; August 1–4, 2013; New Orleans, LA.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: SLG and HN were supported in part by the Agency for Healthcare Research and Quality Center for Education and Research on Therapeutics grant (AHRQ CERT 1U19HS021114). The Agency for Healthcare Research and Quality had no role in study design, collection, analysis, and interpretation of data, writing of the report, or the decision to submit the manuscript for publication.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- acute kidney injury
- all-patient refined diagnosis-related groups
- confidence interval
- chronic kidney disease
- International Classification of Diseases, Ninth Revision
- medical record number
- Pediatric Health Information System
- pediatric-modified Risk Injury Failure Loss End stage
- serum creatinine
- Chertow GM,
- Burdick E,
- Honour M,
- Bonventre JV,
- Bates DW
- 5.↵American Society of Nephrology. American Society of Nephrology Renal Research Report. J Am Soc Nephrol. 2005;16(7):1886–1903.
- Ali T,
- Khan I,
- Simpson W,
- et al
- Vachvanichsanong P,
- Dissaneewate P,
- Lim A,
- McNeil E
- Goldstein SL,
- Devarajan P
- Moffett BS,
- Goldstein SL
- Zappitelli M,
- Moffett BS,
- Hyder A,
- Goldstein SL
- Aronsky D,
- Haug PJ,
- Lagor C,
- Dean NC
- Guevara RE,
- Butler JC,
- Marston BJ,
- Plouffe JF,
- File TM Jr.,
- Breiman RF
- Whittle J,
- Fine MJ,
- Joyce DZ,
- et al
- Patrick SW,
- Davis MM,
- Sedman AB,
- et al
- Schaefer MK,
- Ellingson K,
- Conover C,
- et al
- Tieder JS,
- Hall M,
- Auger KA,
- et al
- Harris JM II.,
- Gay JC,
- Neff JM,
- Patrick SW,
- Sedman AB
- 24.↵3M Health Information Systems. All Patient Refined Diagnosis Related Groups (APR-DRGs) Version 20.0 Methodology Overview. 2003. Available at: www.hcup-us.ahrq.gov/db/nation/nis/APR-DRGsV20MethodologyOverviewandBibliography.pdf. Accessed December 13, 2013.
- Goldstein SL,
- Kirkendall E,
- Nguyen H,
- et al
- Zappitelli M,
- Parikh CR,
- Akcan-Arikan A,
- Washburn KK,
- Moffett BS,
- Goldstein SL
- Soler YA,
- Nieves-Plaza M,
- Prieto M
- 32.↵Agency for Healthcare Research and Quality. Quality Indicators. 2013. Available at: www.qualityindicators.ahrq.gov. Accessed December 13, 2013.
- 33.Centers for Medicare and Medicaid Services. Medicare Hospital Compare Quality of Care. 2013. Available at: www.medicare.gov/hospitalcompare. Accessed December 13, 2013.
- 34.↵Centers for Medicare and Medicaid Services. Statute Regulations Program Instructions. 2012. Available at: www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/HospitalAcqCond/Statute_Regulations_Program_Instructions.html. Accessed December 13, 2013.
- Copyright © 2014 by the American Academy of Pediatrics