Background: The problem list is a meaningful use incentivized criterion, and >80% of patients should have 1 problem entered as structured data.
Objective: The aim of the present study was to use a series of interventions to increase the use of the problem list for inpatients to >80% as measured by at least 1 hospital problem at discharge.
Methods: This study was a quasi-experimental time series quality improvement trial. The primary outcome was 80% of medical and psychiatric inpatients with a problem added to the problem list before discharge. Control charts of percentage (p charts) of medical and psychiatric patients with an inpatient problem list at discharge were constructed with three-σ control limits. Control limits were revised after evidence of improvement. The charts were annotated with interventions, including increasing awareness, focused education, and timely feedback in the form of performance graphs e-mailed to providers.
Results: For medical inpatients, use rose from 31% to 97% at its peak in April 2011 and continues to maintain above the goal of 80%. In psychiatry, problem list use rose from 2% initially to an average of 72% after the interventions.
Conclusions: Significant gains were made with inpatient problem list usage by the medical and psychiatric teams. Our goal ascribed by meaningful use for >80% of inpatients to have a problem at discharge was met after initiation of our series of interventions.
- care standardization
- electronic medical record
- electronic problem list
- information technology
- problem list
- problem lists
- quality improvement
Problem lists are an essential part of the medical record. Dr Lawrence Weed was the first physician to emphasize organization of the patient’s medical chart around the patient’s problem list via problem-oriented charting.1 The medical record when organized in this format improves clinical teaching and encourages clear documentation within the record to improve communication.2 The American Health Information Management Association Best Practice Guidelines for Problem List in an electronic health record (EHR) defines the problem list as “a compilation of clinically relevant physical and diagnostic concerns, procedures, and psychosocial and cultural issues that may affect the health status and care of patients.”3
Today, the problem list is an incentivized criterion of the Health and Information Technology for Economic and Clinical Health Act as part of “meaningful use” of the EHR.4 For stage 1 of meaningful use, it is core measure 3 for both eligible hospitals and eligible providers and states as the objective, “Maintain an up-to-date problem list of current and active diagnoses.” It is measured as, “More than 80 percent of all unique patients admitted to the eligible hospital or critical access hospital (CAH’s) inpatient or emergency department have at least one entry or an indication that no problems are known for the patient recorded as structured data.”5
Accurate problem lists can help guide care and assist in clinical decision support for the physician.6 Problem lists are valuable in helping clinicians do the following: identify the most important health issues for a patient, identify a specific disease population within an electronic record, to enlist certain patients into research studies. They are also beginning to be used for physician and hospital performance.7 Although problem lists are generally accepted as an effective way to organize safe, effective patient care, there are challenges to physicians keeping accurate problem lists. In a recent study assessing physician attitudes that contribute to incomplete problem lists, a variety of themes emerged: “workflow, ownership and responsibility, relevance to practice, uses, content, presentation, accuracy, alternatives, support/education, and culture.”8
Several studies have been conducted concerning problem list use. Most studies to date have focused on standardizing language within the problem list,9 natural language processing as a means of capturing problems written by the physician in a note to import to the problem list,10–12 forcing physicians to enter a diagnosis for which medications were ordered in the inpatient setting to populate the problem list,13,14 or building alerts around diagnostic rules that notify physicians to add particular problems to the list as a means of generating a problem list.15 To the best of our knowledge, no studies have been conducted on improving use of the problem list itself without reliance on other techniques to automatically add problems to the problem list.
Before this study, problems lists were not frequently maintained on inpatients in our hospital. As part of a larger focus of improving electronic documentation within our hospital and as a means to improve problem list usage for meaningful use, we implemented a quality improvement project to increase use of the problem list. The aim of the present study was to use a series of interventions to increase the use of the problem list for inpatients to >80% as measured by at least 1 hospital problem at discharge.
The Children’s Hospital Colorado is a 318-bed tertiary care children’s hospital with 13 557 inpatient admissions annually. The average length of stay is 5.7 days.16 Children’s Hospital Colorado was the first children’s hospital in the United States to fully implement a single EHR (Epic, Epic Systems, Madison, WI) throughout the continuum of care. We used a comprehensive computerized record, as described in the literature, beginning in 2007 when computerized provider order entry was implemented.17
The teams involved in the study included 2 general medicine hospitalist teams, a nurse practitioner–led team, 2 specialty teams, and the psychiatric team (Table 1). Each physician-led team cares for ∼20 inpatients; the nurse practitioner team has fewer providers and thus carries fewer patients. The interventions were spread to the psychiatry inpatient services after initial success with the medical inpatient teams.
Beginning in April 2010, a series of educational interventions were implemented. We first started transitioning from a system-based documentation style, similar to the documentation strategy used in critical care in which each organ system is documented along with the details of the plan that pertain to that system, to a problem-based approach in which pneumonia would be listed rather than pulmonary and infectious disease as the heading for the management plan details. Both hospitalists and the residents served on the documentation committee, and feedback was solicited from both groups. In addition, we wanted to separate the hospital course from the daily progress note management plan (it was blended together in the resident daily progress note before this intervention). We enlisted a physician analyst regarding a different note type. This method located the overall hospital course within the problem list and by problem rather than in the daily progress note. If updated, the hospital course on each problem would automatically pull to the discharge summary, transfer summary, or interim summary instead of having to be rewritten in a note. We used this method as incentive for continued problem list use.
Based on the initial results of our team A pilot, the following interventions occurred with these basic structure, awareness and teaching/education methods followed by structured feedback on problem list use by the team shared with all team members and between teams: (1) training on the problem list was added to resident and fellow training for the upcoming academic year; (2) step action handouts were created and distributed to the pediatric residents; (3) during July 2010, key stakeholders (including the section heads, faculty, and house staff) were made aware of the plan to begin using problem lists for all inpatients; (4) in October, each medical team received focused teaching on rounds regarding use of the problem list; and (5) e-mail usage graphs were initiated biweekly in mid-October to give real-time feedback to the teams regarding their daily problem list use. Biweekly, the usage percentage for each team was calculated for each day and sent out on a graph via e-mail to the entire hospitalist faculty, the residents, and interns for each team for all shifts, all nurse practitioners on team D, and the attending on service for teams C and E. We wanted to stimulate spread and sustainability. Based on work in our hospital toward creating physician “data dashboards,” this step was used as a preliminary option to see how physicians would react to data regarding team problem list use and comparison with other teams in the hospital.
Once we were able to demonstrate success, by January 2011, the problem list was spread to the psychiatry unit by using similar strategies. An initial meeting was held with the faculty of the section showing them use of the problem list and answering questions. The section was then added to the problem list usage graphs in February 2011.
This study was reviewed by the institutional review board for human subjects and was deemed exempt. No informed consent or further review was required because this study was considered an ongoing quality improvement project.
We used a quasi-experimental, time series design18 to determine if problem list usage improved after introducing multiple different interventions in series. Secondary to meaningful use guidelines set forward by the Health and Information Technology for Economic and Clinical Health Act, the goal was >80% of medical and psychiatric inpatients with a hospital problem entered before discharge.4 The interventions were initially piloted on team A and then spread to the 4 different medical teams caring for the inpatients within our hospital. Two months of baseline data served as the control for the entire medical inpatient group.
Data collection occurred from February 2010 to October 2011. Problem list use, as defined as the primary outcome of this study, was the entry of one hospital problem to the problem list prior to discharge of the patient from the hospital. A monthly problem list usage report was obtained; it assessed how many children had a hospital problem noted on the problem list at time of discharge. Percent use for each medical team, as well as all medical inpatients, was determined. Percent use was also determined for the psychiatry team.
Before starting the intervention of using the problem list, baseline data were obtained from February 2010 to March 2010 (Table 1). The number of hospital problems noted at discharge was compared with the number of patient discharges per team.
Using monthly percent problem list use at discharge by team, a run chart and a statistical process control chart (a p chart), with three-σ control limits, were constructed; the p chart allowed us to differentiate between variations intrinsic to the process being measured (common cause variation) versus variation that occurs beyond the expected process variation (special cause variation). Control limits were revised with evidence of improvement in the process. The control chart was annotated with initiation of interventions to note their impact on improvement. The data were also stratified according to team. The same was done for the psychiatry section. The software used to calculate the p charts and control limits was QI CHARTS version 1.0.44 (Scoville Associates, Raleigh-Durham, NC).
Problem List Use by Team
As the pilot hospitalist team, team A started with a 9% use of hospital problem list at patient discharge. This rate increased to 91% in October 2011. For team B, use rose significantly after the interventions but then dropped to 76% near the end of the study with a decrease in performance in September and October 2011. Team C had a pattern similar to team B with initial improvement but a drop to 69% at the end of the study period. Team D was the best, with overall use after the educational period: 100% use was achieved for 4 different months. Team E’s improvement was more moderate, with use still dipping below 80% after the educational interventions were completed (Fig 1). Overall, total medical team use rose from 27% to 97% at its peak in April 2011, and it continued to be maintained above the goal of 80% (Fig 2). In all areas, use improved to the point of adjusting the control limits upward for the sustained improvement.
Problem List Use–Psychiatry
After the improvement occurred with the medical teams, the educational training was spread to the psychiatry team, although the notes did not change for this group of providers. An initial educational meeting occurred in early January 2011. The psychiatry team was added to the biweekly e-mail graph of problem list use on February 1, 2011. Problem list use rose from 3% initially to an average of 72% after the intervention. This group also experienced a slight decline (from 75% to 59%) at the end of the study period (Fig 3).
All areas experienced marked improvement in problem list use after initiation of our interventions. Our goal ascribed by meaningful use for >80% of inpatients to have a problem at discharge was met after initiation of our series of interventions. Overall, use remained >80% after all interventions were in place. Improvement was noted after key stakeholder awareness was completed and continued to rise with focused education and was sustained with biweekly e-mail graphs to providers regarding their performance. The interventions were implemented in series, but the limited time between each implementation restricts our ability to determine with certainty which interventions affected the outcome.
There were a number of limitations to this study. First, although improvement seemed sustained after the implementation of daily problem list e-mail graphs, we are unable to determine (without discontinuing the e-mail graphs) whether the improvement would have been sustained without these graphs. Second, we do not have data on the accuracy of the problem list; therefore, we are unable to comment on the accuracy and can only speak to use of at least 1 problem being added during the inpatient stay as noted at discharge. Third, there was a lack of accurate assignment of patients in the monthly reports to the correct team, leaving many in a “general” category rather than being placed in the correct team. This issue affected the number of discharges reported for each team in the monthly reports and could have led to error, although it likely skewed the data toward the null hypothesis. Several initiatives have been enacted to correct this issue and better identify patients with the team they were assigned to for reporting purposes. Fourth, it is difficult to determine if these interventions would generalize to other subspecialties (specifically the surgical services) or to a service that does not have residents doing the bulk of the documentation. Finally, because our study was conducted at a single site, it is also difficult to know if these interventions would work outside of a children’s hospital environment.
Overall, we were able to demonstrate improvement in inpatient medical and psychiatric patient problem list use by provider teams without reliance on automation of the problem list with our series of interventions. This finding suggests providers are able to maintain a problem list without one being automatically generated through other work-around techniques or forcing entry of problems into the problem list via another route.
First, these data provide information on problem list use, but there is no way to ensure their accuracy from this study. A follow-up study to look at problem list accuracy within the EHR is a necessary next step. Second, this study suggests that physician dashboards may be a potential tool for driving physician performance around quality of care measures once awareness and education campaigns have taken place. Future studies are needed to describe how providers perceive and assess quality of care data. We have started to study the comparative effectiveness of the data graphs versus an alert built into our EHR as a means of notifying surgeons of patients without an active hospital problem list as the next step in spreading problem list usage throughout the hospital. We are attempting this study without the upfront educational effort to see if just the e-mail data graphs are enough to cause behavior change.
Significant improvement in hospital problem list usage was achieved by using a series of interventions, including increasing awareness, focused education, and timely feedback in the form of performance graphs e-mailed to providers. Real-time distribution of outcome data to frontline staff and various stakeholders may have resulted in sustained improvement in problem list usage, although this outcome is difficult to determine with certainty based on the study design. A drift downward did occur at the end of the study that may have coincided with a waning effect of the previous interventions or initiation of monitoring other services in the hospital simultaneously with the medical and psychiatric teams. Overall, we were able to attain meaningful use stage 1 core measure 3 because mean use was >80% looking at all medical and psychiatric services together without the use of problem list automation to reach this goal.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- electronic health record
- 5.↵EHR Incentive Program. Eligible hospital and critical access hospital (CAH) attestation worksheet for stage 1 of the Medicare electronic health record (EHR) incentive program. Available at: www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/Hospital_Attestation_Worksheet.pdf. Accessed August 3, 2013.
- Meystre S,
- Haug P
- Solti I,
- Aaronson B,
- Fletcher G,
- et al
- Ennis J
- Copyright © 2014 by the American Academy of Pediatrics