“In God we trust, all others bring data.” This pithy quote from Edwards Deming, 1 of the fathers of quality improvement (QI), speaks to the centrality of data and measurement in QI. Early on hospitalists realized that QI in the hospital setting was key to their value proposition and return on investment strategy. However, pediatric hospitalists have long complained about the dearth of nationally recognized inpatient measures, especially ones that drive improvement for patients, create value for the health care system, and are under their control. The predominant pediatric quality collaborative, Solutions for Patient Safety, focuses primarily on patient safety issues, many of which are more pertinent to other specialists (eg, ventilator-associated pneumonia, surgical site infections, obstetrical adverse events),1 and the primary quality metric, readmissions, is controversial in pediatrics.2 Pediatric hospitalists saw the Joint Commission’s Children’s Asthma Care measures as unlikely to improve care secondary to near universal compliance (administering β-agonists and steroids to patients hospitalized for asthma) or lacking evidence and construct validity (providing an asthma action plan).
Recognizing the need for a robust set of QI metrics in the pediatric emergency department (ED) and inpatient settings, Mangione-Smith et al3 endeavored to develop and field test a set of process of care quality indicators, which they term the Pediatric Respiratory Illness Measurement System (PRIMES). They chose conditions with high disease burden and health care costs and found that community-acquired pneumonia (CAP), asthma, bronchiolitis, and croup accounted for 16% of admissions to Pediatric Health Information System hospitals. Using a rigorous process, including formal systematic review of the evidence-base, Mangione-Smith et al3 ultimately proposed 136 draft quality indicators, of which a nationally representative expert panel endorsed 118 as appropriate. The initial measures set was pilot tested at 3 children’s hospitals and 76 (46 for the ED and 30 for inpatient care) were successfully specified. The 76 indicators were then field tested in 3 tertiary care children’s hospitals, and interrater agreement on scores was found to be excellent. Individual indicator and aggregate scores were computed, stratified by site (ED versus inpatient), function (diagnosis, treatment, follow-up), and modality of care (history, physical, laboratory or radiology, medication, ancillary therapy, counseling, referrals, and disposition determination).
Hospital-level summary scores were compared between hospitals and within hospitals across condition, site of care, function, and modality of care. Across the 3 hospitals, 190 to 350 cases per condition were abstracted at each hospital, meeting their predetermined goals for all conditions except croup. Significant between-hospital variation was observed for CAP and croup but not bronchiolitis or asthma. Within-hospital variation was also seen, with ED scores exceeding inpatient scores for asthma and bronchiolitis and vice versa for croup (CAP only contained inpatient indicators). Scores related to treatment decisions and appropriate medication use were the highest, whereas those related to laboratory/radiology testing were among the lowest.
The gargantuan effort put forth by the PRIMES investigators should be recognized and applauded—a comprehensive literature review including 136 draft indicators, broad stakeholder search and inclusion, utilization of the RAND–University of California Los Angeles modified Delphi method,4 including a 2-day in-person meeting, pilot testing, creation of an electronic abstraction tool with automatic scoring, and extensive field testing demonstrating between- and within-hospital variation revealing possible improvement opportunities.
Before we take the next step and implement PRIMES, we must ask some key questions. Do the PRIMES have construct validity (ie, Do they measure what we think they measure and do improved scores equate to greater value?), especially given that 49 of the 76 indicators are based on the lowest level of evidence, expert consensus? Is there enough provider and institution dependent variation that allows for significant improvement driven by measurement and will PRIMES drive that improvement? How will PRIMES perform in community hospitals where most children receive care for these conditions, especially small hospitals with meager sample sizes?
In addition to these questions, we should heed Don Berwick’s admonition to reduce the volume and total cost of mandatory measurement by 50% within 3 years and by 75% within 6 years. “The aim should be to measure only what matters, and mainly for learning.”5 And even though Edwards Deming is credited with saying, “If you can’t measure it, you can’t manage it,” what Deming actually said was, “It is wrong to suppose that if you can’t measure it, you can’t manage it—a costly myth,”6 warning against relying on figures alone, as many important things are unmeasurable yet still must be managed.
Special consideration should be given to the indicators relating to overuse, where clinicians are asked not to do something. An astonishing 30% of the PRIMES (22 of 76) target overuse, an increasingly recognized source of waste and avoidable harm.7 Most QI metrics target underuse, for a variety of psychological and logistical reasons, but targeting overuse has inherent advantages over targeting underuse. Avoiding a nonevidence-based practice has the potential to decrease harm and therefore increase quality, but even in the absence of harm avoidance, inconvenience is usually averted, and costs are invariably decreased. Therefore, compliance with overuse metrics always increases value (defined by quality/costs). The Value in Inpatient Pediatrics network recognizes this advantage, and terms these indicators “value metrics.”8
A reasonable path forward would be to select the 19 indicators supported by the highest level of evidence and include all the value metrics for a total of 31 indicators. These indicators should be trialed in community hospitals. Indicators with near perfect compliance should be removed because they are unable to drive improvement, though they could be considered for accountability or other uses. Goals for indicator scores could be set by applying achievable benchmarks of care, a methodology that uses objective data to derive attainable targets that represent a measurable level of excellence.9 Achievable benchmarks of care should be iterative, changing over time as care improves.
Ultimately, PRIMES has great potential to increase value for pediatric patients presenting to EDs and inpatient units with respiratory conditions. The rigorous development methodology may also serve as a template for other measurement systems. Although the authors are correct when they state, “Future validation work for PRIMES should include formal studies to assess the relationship between high levels of performance on these indicators and other established quality measures such as return to baseline functional status…reduction in return visits to the ED, or fewer 30-day readmissions to the hospital,”3 this new tool is certainly a step in the right direction, and may be ready for prime time.
FINANCIAL DISCLOSURE: The author has indicated he has no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
POTENTIAL CONFLICT OF INTEREST: The author has indicated he has no potential conflicts of interest to disclose.
- Mangione-Smith R,
- et al
- Brook RH
- ↵Closing Keynote Address. 27th Institute for Healthcare Improvement Annual Forum. Orlando, FL. December 9, 2015
- Deming WE
- Ralston SL,
- Garber MD,
- Rice-Conboy E,
- et al
- Parikh K,
- Agrawal S
- Copyright © 2017 by the American Academy of Pediatrics