OBJECTIVES: Drug–drug interactions (DDIs) can result in patient harm. DDI alerts are intended to help prevent harm; when the majority of alerts presented to providers are being overridden, their value is diminished. Our objective was to evaluate the overall rates of DDI alert overrides and how rates varied by specialty, clinician type, and patient complexity.
METHODS: A retrospective study of DDI alert overrides that occurred during 2012 and 2013 within the inpatient setting described at the medication-, hospital-, provider-, and patient encounter–specific levels was performed at an urban, quaternary-care, pediatric hospital.
RESULTS: There were >41 000 DDI alerts presented to clinicians; ∼90% were overridden. The 5 DDI pairs that were most frequently presented and overridden included the following: potassium chloride–spironolactone, methadone–ondansetron, ketorolac–ibuprofen, cyclosporine–fluconazole, and potassium chloride–enalapril, each with an alert override rate of ≥0.89. Override rates across provider groups ranged between 0.84 and 0.97. In general, patients with high complexity had a higher frequency of alert overrides, but the rates of alert overrides for each DDI pairing did not differ significantly.
CONCLUSIONS: High rates of DDI alert overrides occur across medications, provider groups, and patient encounters. Methods to decrease DDI alerts which are likely to be overridden exist, but it is also clear that more robust and intelligent tools are needed. Characteristics exist at the medication, hospital, provider, and patient levels that can be used to help specialize and enhance information transmission.
Adverse drug events (ADEs) can pose significant risks to patients.1–4 Among ADEs are drug–drug interactions (DDIs) in which the coadministration of medications may result in harm by altering medication effectiveness or resulting in toxicity.5 Researchers in a study investigating potential drug–drug interactions (pDDIs) in hospitalized pediatric patients identified that nearly half of the pediatric patients were exposed to a pDDI, with 41% of hospitalizations representing a major pDDI, according to the Micromedex DRUG-REAX System.6
Computerized prescriber order entry systems that provide transactional decision support are aimed at mitigating preventable ADEs, such as DDIs. However, evidence from various clinical settings reveals that too many alerts can result in cognitive overload and alert fatigue, decreasing alert effectiveness. In 1 investigation, 936 medication errors were identified across both inpatient and outpatient settings. Half of all the identified medication errors were considered to be preventable by information technology solutions. Although nearly half of all the errors that resulted in patient harm were due to prescribing errors, the most common cause of errors was found to be the improper use of information technology systems, such as system bypasses.7 There is mounting evidence in the literature that most prescribing alerts presented to providers are overridden. Bryant et al8 demonstrated that in an inpatient setting, 90% of alerts generated from prescriptions were overridden. Weingart et al9 reported that few physicians changed their prescribing behaviors on the basis of the information provided in the alerts. Similarly, high rates of alert override have been reported with drug–allergy alerts10 and in ambulatory settings,11 with up to 92% of alerts being overridden.12 High alert-override rates raise the concern that alerts may be generated too frequently, leading to decreased attentiveness and subsequently improper handling of the information provided.13
Expert consensus groups have suggested avenues for alert modification to enhance current computerized prescriber order entry systems, including developing alerts that are specific to the venue of care, clinical discipline, prescriber knowledge, and patient characteristics.14,15 Despite these recommendations, to our knowledge, there are no studies about the circumstances in which alerts are generated and overridden. In this study, we intend to further this knowledge by evaluating the contextual (medication type), environmental (venue of care), experiential (provider), and clinical (patient complexity) characteristics associated with frequency of alert presentation and subsequent override by clinicians.
We conducted a retrospective study of DDI alert overrides at the medication-, hospital-, provider-, and patient encounter–specific levels associated with care delivered to patients admitted to the hospital.
The study took place in an urban, pediatric, quaternary-care hospital. The Cerner suite of clinical applications is the electronic medical record in use, with referential data for interactions from Multum.16,17 On the basis of these data, details of the order, and locally specified criteria for alerts, the electronic medical record was used to generate allergy and drug interaction alerts, dose range alerts, and duplicate-order alerts. A screenshot of an allergy alert window is provided in Supplemental Fig 2.
DDI Alerts Sample
Our sample included records of all patients admitted to the hospital for whom a DDI alert was presented to the ordering provider between January 2012 and December 2013. During the study period, the medication system underwent evaluation by a team of clinical, pharmacy, and informatics experts in January 2013. The team identified the DDI pairings with the highest frequency of alert override, and those deemed to have less clinical value for providers were discontinued.
The DDI alert data used in this study were based on interruptions for major contraindicated drugs prescribed during inpatient encounters. The term “major contraindicated” is defined by Multum as an interaction that “poses such a major threat to the patient’s health that it belongs to the highest severity level.”18 Our analysis was focused on the alerts that were most frequently presented to and overridden by clinicians. A total of 33 DDI pairs, accounting for 75% of all alerts presented to the clinicians, were selected for the analysis (Fig 1). Each DDI pair consisted of a trigger medication being newly ordered and a medication already in the order profile or being simultaneously ordered. Because all the medications in the database could be either an existing or a trigger medication, we summed all the combinations for each pair, when applicable, to compute total frequencies.
The National Association of Children’s Hospitals and Related Institutions relative cost weights associated with the 3M All Patient Refined Diagnosis Related Group version 24 and the severity of illness categories assigned to them at discharge (minor, moderate, major, or extreme) were used to account for medical complexity and severity of illness. Severity of illness assignment is influenced by numerous variables, including secondary diagnoses, patient age, secondary procedures, and birth weight. We accounted for these variables using the Case Mix Index (CMI) as a proxy for patient severity and resource use.19 We categorized total National Association of Children’s Hospitals and Related Institutions children’s hospital relative length of stay (LOS) weight ≤1 as “low CMI,” between 1 and 2 as “medium CMI,” and ≥2 as “high CMI.”
DDI alert overrides that occurred during hospital stays during the 2012 and 2013 calendar years were included.
Alerts presented to nonprescribers (pharmacists and medical students), duplicative alerts (eg, generated when a prescriber was writing an order for an already-administered medication), and chemotherapeutic agents used as a part of oncology treatment (when a separate, custom, computerized chemotherapy ordering system and process was used) were excluded.
We summarized DDI alerts presented and overridden on the following characteristics: medication-specific level by DDI pairing, provider-specific level by provider type, provider reason for alert override for alerts associated with inpatient encounters, patient encounter–specific characteristics for alerts associated with inpatient encounters by the specialty to which the patient was admitted, and average number of alerts generated per patient encounter.
We evaluated all DDI alerts in our system for frequency of alert presentation, frequency of alert override, and rate of override. These pairings were evaluated to assess for differences based on provider type (residents, attending physicians, fellows, and nurse practitioners [NPs] and/or physician assistants [PAs]), override reason (treatment plan requirement, benefits outweigh risks, alert not applicable, and other [please document, high or low dose appropriate and infusion instructions modified]), clinical specialty, and patient complexity level (low, medium, and high). Specialty data were evaluated to account for inpatient clinical workflows.
For data presentation purposes, we report the results of the 5 DDI pairings that were most frequently overridden in our system. These pairs accounted for more than half of all the alerts presented and overridden in the system during 2012 and 2013. For these top 5 DDI pairs, the rates of DDI alert overrides were calculated as a ratio of alerts overridden to alerts presented. For the patient encounter–level analysis, we calculated alert override rates per encounter, per patient, and per patient day for inpatient encounters. We calculated patient days from the LOS for the associated encounters during the study period. Differences in alert override rates across provider types, service areas, override reasons, and patient characteristics were compared by using the Kruskal-Wallis test.
The study protocol was approved by the institutional review board. All statistical analyses were performed by using SAS version 9.3 (SAS Institute, Inc, Cary, NC).
During the study period, 1 739 680 medication orders were placed, and 88 321 alerts were associated with hospital admissions. Of the total 88 321 DDI alerts, 46 850 (53%) were excluded from our analysis (Supplemental Fig 3), leaving a total of 41 471 DDI alerts for analysis. Of these, 36 988 were overridden for an alert override rate of 0.89. These alerts were presented to 1747 prescribers across 5221 hospital admissions with a median encounter length of 3.3 days (interquartile range [IQR]: 1.1–8.8). We present the sample characteristics in Table 1.
DDI Pair–Level Analysis
The top 5 DDI pairings that were most frequently overridden over the 2-year period included potassium chloride–spironolactone (0.94 [6532 alerts overridden, 6983 presented alerts]), methadone–ondansetron (0.93 [2583 alerts overridden, 2785 presented alerts]), ketorolac–ibuprofen (0.95 [2051 alerts overridden, 2149 presented alerts]), cyclosporine–fluconazole (0.89 [1605 alerts overridden, 1797 presented alerts]), and potassium chloride–enalapril (0.93 [1277 alerts overridden, 1369 presented alerts]; Supplemental Table 5). Override rates for the remaining DDI pairs ranged between 0.70 (metoclopramide–prochlorperazine) and 0.99 (ziprasidone–haloperidol).
There were 1747 providers included in the provider-level analysis over the 2-year study period. Provider types included residents, fellows, NPs, PAs, and attending physicians. Alert override rates were similar across provider types. Residents were most frequently presented with alerts and had a similar frequency of alert override (0.88 [16 806 alerts overridden, 19 042 alerts presented]) compared with that of NP and/or PAs (0.90 [8894 alerts overridden, 9865 alerts presented]), attending physicians (0.89 [5446 alerts overridden, 6097 alerts presented]), and fellows (0.92 [4680 alerts overridden, 5078 alerts presented]). For the top 5 most frequently overridden DDI pairings, alert override rates were similarly high among the provider types (0.84–0.97; Table 2).
Alerts were most frequently presented to providers in cardiology and/or cardiology services (override rate of 0.91); hematology, oncology, and/or bone marrow transplant (BMT) (override rate of 0.90); and intensive care services (override rate of 0.88), each with >5000 alerts presented to providers during the study time period. We found that >85% of alerts for medications prescribed in these clinical areas were overridden. Comparatively, rheumatology (override rate of 0.77), neurosurgery (override rate of 0.78), and gastroenterology (override rate of 0.81) had the lowest alert override rates (0.77–0.81), with <1000 alerts presented per clinical service during the study time period. However, we observed variability across service lines in both the frequency of alert presentation and alert override rates (Table 3). The complete list of alert override rates by specialty can be found in Supplemental Table 6.
Alerts for the top 5 DDI pairings in our analysis had alert override rates that were >0.85 and did not vary significantly among specialties. The highest frequency of alert overrides occurred with medications that are commonly used by distinct specialties. For example, the override rate for methadone–ondansetron alerts was 0.94 (881 alerts overridden, 942 alerts presented) among clinicians caring for patients on the hematology, oncology, and/or BMT services, whereas the override rate for potassium chloride–spironolactone was 0.94 (5901 alerts overridden, 6280 alerts presented) among clinicians caring for patients in cardiology and/or cardiology services. Meanwhile, the lowest rates of alert overrides occurred most often among providers who prescribed medications less frequently. Detailed information on DDI alert overrides by specialty is summarized in Table 3.
The system specifications require providers to document the reason for overriding an alert. Providers were offered an option to select an override reason from the drop-down menu (Supplemental Table 7) or record a reason in free-text form. The 3 most frequently selected reasons to override DDI alerts were “treatment plan requirement” (override rate of 0.45 [11 700 alerts overridden, 26 972 alerts presented]), “benefits outweigh risks” (override rate of 0.35 [9520 alerts overridden, 26 972 alerts presented]), and “alert not applicable” (override rate of 0.10 [2832 alerts overridden, 26 972 alerts presented]). These 3 reasons accounted for 89% of all the reasons selected during the study period. Across the 5 DDI pairings, there is variation in the frequency of selecting treatment plan requirement and benefits outweigh risks (Supplemental Table 8).
Encounter- and Patient-Level Results
An analysis of patient-level data of alerts presented and overridden in inpatient care settings revealed that there were 4172 unique patients who experienced a total of 5216 inpatient encounters during the study period, with a median encounter lasting 3.3 days (IQR: 1.1–8.8 days). Of the 4172 patients, there were 1679 with high complexity patients, 1788 with medium complexity, and 705 with low complexity. Patients with an LOS >10 days had higher rates of alert overrides per encounter than patients with a shorter LOS (4.7 alert overrides per encounter for patients with an LOS <5 days compared with 12.3 alert overrides per encounter for patients with an LOS >10 days). There were 29 201 alerts presented and 26 678 overridden for an average of 8.1 overridden DDI alerts per encounter. During an average hospital encounter, a DDI alert was generated every 1.9 days and overridden every 2.1 days.
We provide a detailed description of DDI alert overrides and patient complexity in Table 4. These DDI alert overrides occurred more frequently in patients with high complexity, with the potassium chloride–spironolactone interaction alert being overridden an average of 18.9 times per encounter in these patients (Table 4). DDI alert override rates were compared among patient complexity levels and were not statistically significant.
We examined the burden of alerts presented and overridden by providers in 1 health care system and confirmed past findings that DDI alerts are frequent and that the majority are overridden. This finding held across provider level of training, service line to which the patient is admitted, and potential severity of DDI.
Hospital-, provider-, encounter-, and patient-specific characteristics can influence the risks and benefits of medication prescribing. Previous researchers identified differences in clinician medical decision-making and clinical expertise with alert handling20; we did not see differences in alert override rates among clinician types. However, the need for specialty- and context-specific presentation of alerts is reflected in our data by the high percentages of alerts that are overridden for medications that are commonly used by specialists or within specialty areas. In light of these data, we reinforce the recommendations of an expert panel to maintain general consistency of alert content across general providers and specialists while fine tuning that information to best fit the context-specific needs of the providers.15
Integration of patient-specific information, such as laboratory values, severity of illness, the location of patient care, and experiential customization, have been suggested as essential to making alerts more targeted.14,21 Including patient-specific information in the alert generation algorithms may help address the issue of alert redundancy during recurrent care encounters, which was evidenced in this investigation when medication-specific alerts, both within and among patient encounters, were repeatedly generated.22
Importantly, patient complexity should be considered when creating and modifying alerts. Patients with high complexity are a vulnerable population in our health system and are often characterized by the involvement of multiple clinicians in their care, increased frequency of care episodes, and the high volume of medications prescribed.22–25 There is inherent risk associated with caring for patients with high complexity and their polypharmacy that can be equally complex to manage. We observed variability in the number of alert overrides per encounter that correlates with patient complexity levels. This finding is not unexpected because the aforementioned polypharmacy is likely to lead to a greater number of drug interactions and higher alert rates in patients with higher complexity. However, it is essential to take into account patient-level factors, including complexity, when designing efficient and safe medication order systems.
This study was not without limitations. Alert overrides and alert override rates are 1 aspect of investigating the efficacy of alert presentation. Other markers that are not investigated here include specifically analyzing the alert dwell time,26 alert salience, and comparisons of alert overrides with ADEs.27 These factors are potential future avenues for study.
We presented data on how systematic evaluation at the level of the medication, the DDI pair, the provider, the specialty, and the patient encounter could all be used by organizations to develop optimization strategies for DDI alerts. Opportunities to adjust alert rules are clearly constrained by and dependent on individual vendor functionalities. Consequently, our findings should be applied judiciously in initiatives to improve medication order systems. Despite enhancements to the medication order system that can be done locally, a considerable alert burden still exists. To increase alert value, new functionality will be needed to further decrease alert frequency and improve the clarity and usefulness of the information delivered within the alerts.9
Rates of DDI alert overrides are high across provider groups, service lines, and medications and vary according to patient complexity. Enhancing the specificity and sensitivity of the alerts presented to providers is essential to improving how key information is communicated to clinicians. Opportunities exist at the medication, hospital, provider, and patient levels to enhance information transmission but will need further investigation and refinement over time.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported through grant funds issued by The Risk Management Foundation of the Harvard Medical Institutions (CRICO/RMF).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Institute of Medicine. Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: National Academies Press; 2012
- Brunton L,
- Chabner B,
- Knollman B
- Feinstein J,
- Dai D,
- Zhong W,
- Freedman J,
- Feudtner C
- Stultz JS,
- Nahata MC
- Yeh ML,
- Chang YJ,
- Wang PY,
- Li YC,
- Hsu CY
- Harper MB,
- Longhurst CA,
- McGuire TL,
- Tarrago R,
- Desai BR,
- Patterson A
- ↵Multum [Internet]. Denver, CO: Cerner Corporation. Available at: http://www.cerner.com/multum_solutions/. Accessed March 19, 2018
- ↵Cerner Knowledge Sharing Platform. Alert severity levels in powerorders. Available at: https://wiki.ucern.com/display/public/1101powerordersHP/Alert+Severity+Levels+in+PowerOrders. Accessed November 30, 2016
- Sedman AB,
- Bahl V,
- Bunting E,
- et al
- Cho I,
- Slight SP,
- Nanji KC,
- et al
- Copyright © 2018 by the American Academy of Pediatrics