Development and Validation of the Pediatric Medical Complexity Algorithm (PMCA) Version 2.0
BACKGROUND AND OBJECTIVES: The Pediatric Medical Complexity Algorithm (PMCA) was developed to stratify children by level of medical complexity. We sought to refine PMCA and evaluate its performance based on the duration of eligibility and completeness of Medicaid data.
METHODS: PMCA version 1.0 was applied to a cohort of 299 children insured by Washington State Medicaid with ≥1 Seattle Children’s Hospital outpatient, emergency department, and/or inpatient encounter in 2012. Blinded assessment of the validation cohort’s PMCA category was performed by using medical records. In-depth review of discrepant cases was performed and informed the development of PMCA version 2.0. The sensitivity and specificity of PMCA version 2.0 were assessed.
RESULTS: Using Medicaid data, the sensitivity of PMCA version 2.0 was 74% for complex chronic disease (C-CD), 60% for noncomplex chronic disease (NC-CD), and 87% for those without chronic disease (CD). Specificity was 84% to 91% in Medicaid data for all 3 groups. Medicaid data were most complete for children that had primarily fee-for-service claims and were less complete for those with some managed care encounter data. PMCA version 2.0 performed optimally when children had a longer duration of coverage (25 to 36 months) with fee-for-service reimbursement, identifying children with C-CD with 85% sensitivity and 75% specificity, children with NC-CD with 55% sensitivity and 88% specificity, and children without CD with 100% sensitivity and 97% specificity.
CONCLUSIONS: PMCA version 2.0 identifies children with C-CD with good sensitivity and very good specificity when applied to Medicaid data. Data quality is a critical consideration when using PMCA.
In March 2011, the Pediatric Quality Measures Program was mandated by the 2009 Child Health Insurance Program Reauthorization Act1 and charged with the development of new and/or enhancement of existing quality of care measures for children’s health care across the age spectrum.2,3 One aspect of this work was to identify and/or develop a valid methodology to assess disparities in care according to the level of medical complexity for children with special health care needs.4 As the Patient Protection and Affordable Care Act is implemented, Medicaid and the health care system increasingly need strategies to allocate resources; accurate identification of children who are most likely to benefit from care coordination and other resources is critical.5 The Pediatric Medical Complexity Algorithm (PMCA) was developed to facilitate stratification of children by level of medical complexity and was developed and validated by using both Washington State Medicaid (WA-Medicaid) administrative data and hospital discharge data.4 PMCA stratifies children into 3 levels of chronic disease (CD): complex chronic disease (C-CD), noncomplex chronic disease (NC-CD), and without CD. Among children insured by WA-Medicaid who accessed tertiary hospital inpatient or emergency department (ED) care, PMCA version 1.0 exhibited very good sensitivity (84%) for correctly categorizing children with C-CD, excellent sensitivity (96%) for correctly categorizing children without CD, but poor sensitivity (41%) for correctly categorizing children with NC-CD.4
After distribution to >40 organizations and researchers in a wide variety of health care systems and data sources (Supplemental Table 4), most users have described PMCA version 1.0 as relatively easy to implement. However, some users reported concerns about the inclusion and/or exclusion of various conditions and their accompanying International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes (eg, users recommended inclusion of morbid obesity as a nonprogressive chronic condition) and the classification of particular conditions as progressive, thus denoting a condition associated with a shortened life expectancy (eg, users recommended removing the progressive flag for hypercholesterolemia).
We recognized an opportunity to additionally refine the algorithm by applying PMCA version 1.0 to a new validation cohort of children with Medicaid coverage by performing blinded review and assignment of PMCA categories in the validation cohort by using medical records review and conducting additional in-depth review of discrepant cases. The objectives of the current study were to (1) refine PMCA, and (2) evaluate the performance of PMCA version 2.0 based on the duration of eligibility and completeness of Medicaid data.
Algorithm refinement consisted of 4 stages: (1) application of PMCA version 1.0 to create a validation cohort of children with Medicaid coverage, (2) blinded review by 1 author (J.P.) of the validation cohort’s medical records with assignment to PMCA category, (3) in-depth review of discrepant cases by using multiple data sources by 2 authors (T.D.S. and R.M.S.), and (4) development of PMCA version 2.0 based on these reviews. The evaluation focused on the sensitivity and specificity of PMCA version 2.0, as well as the sensitivity of PMCA version 2.0 to the duration of eligibility and completeness of Medicaid data. All study procedures were reviewed and approved by the Institutional Review Boards of Seattle Children’s Research Institute and Washington State.
Application of PMCA Version 1.0 to Create a Medicaid-Insured Validation Cohort
Children 3 months to 18 years of age, insured by WA-Medicaid, and who received outpatient or ambulatory, ED, and/or inpatient services at Seattle Children’s Hospital (SCH) in 2012 were potentially eligible for the study. As recommended for use with claims data, the more conservative version of PMCA version 1.0 was applied to 3 years (2010 through 2012) of WA-Medicaid administrative data.4 After PMCA version 1.0 categorization, a block-randomized sample of 300 children, including 100 in each PMCA category, was obtained. We selected 300 children to ensure adequate numbers in each PMCA category. Almost all WA-Medicaid children were successfully matched in the SCH data. However, because we were unable to find encounters for 1 child in the SCH medical records system, the validation cohort included 299 children.
Blinded Review and PMCA Assignment by Using the Validation Cohort’s Medical Records
A trained nurse researcher (J.P.), blinded to PMCA categorization, made assignments into 1 of 3 levels of CD by reviewing all available SCH electronic medical records, including outpatient, ED, and inpatient encounters from any time period. The assignment of CD level relied on the previously developed consensus definitions.4 In 117 cases, when the initial CD level assignment was unclear for the nurse researcher, the medical records were reviewed a second time by 2 physicians (T.D.S. and R.M.S.), who were also blinded to PMCA categorization, and assignments were made by consensus. Examples of cases with unclear initial CD level assignment include conditions that resolved after surgical intervention (eg, ear infections that resolved after tympanostomy placement) and conditions with limited impact on function (eg, a diagnosis of vitiligo but no care sought for >1 year). In 8 (7%) of the 117 cases discussed, the CD level assignment was changed from the one assigned initially by the nurse researcher.
In-Depth Review of Discrepant Cases
There were 85 cases with discrepancies between PMCA assignment and medical records review in the validation cohort. For every discrepant case, the PMCA assignment, the ICD-9-CM codes used to make that assignment, and the child’s medical record were reviewed independently and summarized by 1 of 2 physicians (T.D.S. and R.M.S.). Each case was then discussed among the 2 physician reviewers, with careful consideration of the reasons for misclassification. One example of a case with misclassification was a child who was noted to have bilateral sensorineural hearing loss by medical record review and was thus classified as having a NC-CD; whereas, in contrast, the claims data indicated hearing loss (ICD-9-CM codes 389.1, 389.2, 389.9), adjustment reaction (ICD-9-CM code 309.9), and hemiplegia (ICD-9-CM code 438.2) resulting in assignment to the C-CD category.
Development of PMCA Version 2.0
The in-depth review of discrepant cases resulted in the inclusion of additional conditions and their accompanying ICD-9-CM codes as well as the removal of some ICD-9-CM codes included in the original PMCA version 1.0. The review additionally resulted in changes to some of the original progressive flags (ie, flags for conditions that were deemed to shorten a child’s life expectancy to ≤40 years) assigned in PMCA version 1.0. See Supplemental Table 5 for all ICD-9-CM code changes made in PMCA version 2.0.
Evaluation of the Sensitivity and Specificity of PMCA Version 2.0
We determined the sensitivity and specificity of PMCA versions 1.0 and 2.0 for correctly classifying children into the 3 levels of complexity using WA-Medicaid administrative data on the same population of 299 children. The sensitivity of each PMCA version was calculated by using the number of correctly identified children for each of the 3 levels (true-positives) divided by the total number in the level (true-positives and false-negatives). For example, the sensitivity of PMCA version 1.0 for C-CD was 70%, with 71 correctly identified children with a C-CD out of 101 total children with a C-CD (Table 1). Similarly, the specificity of each PMCA version was calculated by using the number of correctly not-identified children for each of the 3 levels (true-negatives) divided by the total number of not-identified children in the level (false-positives and true-negatives). For example, the specificity of PMCA version 1.0 for C-CD was 85%, with 169 (19 + 68 + 75 + 7) correctly not identified as C-CD out of 198 total (112 + 86) not identified as C-CD (Table 1). Because there are no universal cutoffs for sensitivity and specificity,6 we decided a priori criteria for excellence was >90%, very good was 80% to 90%, and good was 70% to 80%.7
During the study period, fee-for-service Medicaid claims data tended to have more complete administrative data compared with Medicaid managed care encounter data provided to Washington state.8 Therefore, we also evaluated the performance of PMCA version 2.0 in the validation cohort by duration of eligibility and completeness of Medicaid data. Any gaps in Medicaid coverage >1 month were captured. WA-Medicaid data included all inpatient and outpatient administrative data provided to the state, either by fee-for-service in the form of claims or managed care health plans in the form of encounter data. Within the 3-year study period, both eligibility and type of coverage were classified into 1 to 4 months, 5 to 12 months, 13 to 24 months, and 24 to 36 months. The number of months of each type of coverage each eligible child contributed (n = 217 for fee-for-service and n = 267 for managed care health plans) was considered.
Review of the 85 discrepant cases in the validation cohort shown in Table 1 generated several changes to PMCA version 2.0. The final revised algorithm constitutes PMCA version 2.0 (Supplemental Table 6).
In the sample of 299 children in the validation cohort, PMCA version 1.0 identified children with C-CD with 70% sensitivity and 85% specificity, children with NC-CD with 61% sensitivity and 83% specificity, and children without CD with 87% sensitivity and 89% specificity (Table 1). Modifications resulted in minimal improvement in performance, because PMCA version 2.0 identified children with C-CD with 74% sensitivity and 84% specificity, children with NC-CD with 60% sensitivity and 84% specificity, and children without CD with 87% sensitivity and 91% specificity (Table 2).
PMCA version 2.0 performed optimally when children had a longer duration of coverage (25 to 36 months) with more complete fee-for-service claims data, with the modified algorithm identifying children with C-CD with 85% sensitivity and 75% specificity, children with NC-CD with 55% sensitivity and 88% specificity, and children without CD with 100% sensitivity and 97% specificity (Table 3). In contrast, when children had a similar duration of managed care coverage, the modified algorithm identified children with C-CD with 73% sensitivity and 86% specificity, children with NC-CD with 64% sensitivity and 85% specificity, and children without CD with 86% sensitivity and 89% specificity.
This article describes refinements made to PMCA by applying PMCA version 1.0 to a new validation cohort of children with Medicaid coverage, performing blinded review and assignment of PMCA categories in the validation cohort, and conducting chart review of discrepant cases, which resulted in the development of PMCA version 2.0. Modifications to PMCA version 2.0 resulted in improved face validity, but minimal improvement in performance compared with PMCA version 1.0, so we recommend routine use of version 2.0. SAS code and documentation for PMCA version 2.0 are available for download at www.seattlechildrens.org/research/child-health-behavior-and-development/mangione-smith-lab/measurement-tools/.
We found that the quality of administrative data impacts the performance of PMCA, and it is critical for PMCA users to understand the impact of the quality and completeness of the data being used on its performance. We demonstrated better performance of PMCA version 2.0 when children had more complete fee-for-service claims data compared with less complete managed care encounter data provided to WA-Medicaid during the study period. With a minimum of 5 months of fee-for-service coverage, PMCA version 2.0 exhibited very good sensitivity for correctly categorizing children with C-CD and very good sensitivity for correctly categorizing children without CD, whereas performance was less optimal with comparable managed care coverage. PMCA users with high quality and complete data (such as WA-Medicaid fee-for-service claims) can be reassured of the algorithm’s optimal performance. However, PMCA users with less complete data (such as WA-Medicaid managed care encounter data during this study period) or rapid turnover in their insured population should be aware of the algorithm’s limitations in correctly classifying children under these circumstances. One approach described in our earlier work to hospital discharge data, which are even less complete than managed care encounter data, is to use the least conservative version of the PMCA code.4 However, this approach can increase the likelihood that false-positive findings would be observed in the C-CD group because only 1 claim for each of 2 separate body systems is required during the measurement period for classification into this group. This approach potentially results in children who have a history of chronic disease in 2 body systems, but resolved disease in at least 1 (NC-CD) body system being incorrectly classified by PMCA as having C-CD.4 All PMCA version 2.0 users should carefully consider the completeness of their source data when selecting which analytic approach (the least or more conservative approach) to use.
As we observed in our earlier work with PMCA version 1.0,4 PMCA version 2.0 continues to perform poorly in identification of children with NC-CD in administrative data. These conditions may be episodic for a given child and can encompass a wide range of severity, from mild to severe. This variation results in highly fluctuating health care use over time within the NC-CD group. Because PMCA was applied to Medicaid administrative data, health care use both within and outside of SCH was available to inform stratification. Portions of a child’s care occurring in settings outside of SCH were not available to the medical records reviewer. It is possible, therefore, that relevant conditions were not captured when assigning PMCA category to children in the validation cohort; however, we carefully reviewed medical history in an attempt to capture conditions managed outside of SCH.
The current study had several additional limitations. Approaches relying on secondary data containing ICD-9-CM codes are only as good as the completeness and accuracy of the original data sources. Therefore, children who do not interact with the health care system will lack the necessary information to define their level of medical complexity, and children enrolled in ≥2 health plans will have incomplete encounter information in both data sets unless the data sets are merged. Furthermore, encounters for conditions that do not result in a claim, or claims not submitted by managed care organizations to Medicaid, will not be captured in Medicaid administrative data. Any manual assessment of medical records data is subject to human error; however, the independent reviewer for this study is experienced in this type of review4,9 and brought cases where she was unsure to the physician panel for evaluation. We also anticipate the need for additional changes to and additional validation work on PMCA to ensure compatibility with the recent widespread adoption of International Classification of Diseases, Tenth Revision, Clinical Modification codes. These updates are currently underway. However, given that PMCA performance is optimized with 3 years of retrospective administrative data,4 it should likely not be applied to International Classification of Diseases, Tenth Revision, Clinical Modification code data before October 1, 2017.
PMCA version 2.0 is an updated, publicly available algorithm that identified children with C-CD who have accessed tertiary hospital outpatient, ED, or inpatient care, with good sensitivity and very good specificity when applied to Medicaid administrative data, and is similar in performance to PMCA version 1.0. Optimal identification of children with C-CD was observed when using more complete claims data. All PMCA version 2.0 users should carefully consider the completeness of their source data when selecting which analytic approach to use (ie, the least or more conservative approach).4
We thank Dorothy Lyons, MPA and Peter Woodcox of the Research and Data Analysis Division of the Washington Department of Social and Health Services for analyses and coding work.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This work was supported by a cooperative agreement with the Agency for Healthcare Research and Quality, grant U18HS020506, part of the Pediatric Quality Measures Program. Dr Simon was also supported by award K23NS062900 from the National Institute of Neurological Disorders and Stroke, National Institutes of Health. None of the sponsors participated in design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of the funders. Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Zima BT,
- Murphy JM,
- Scholle SH,
- et al
- Simon TD,
- Cawthon ML,
- Stanford S,
- et al
- Cohen E,
- Kuo DZ,
- Agrawal R,
- et al
- Zweig MH,
- Campbell G
- Byrd VL,
- Dodd AH
- Copyright © 2017 by the American Academy of Pediatrics