Problem: Error reports can be used to identify local system hazards, provide analysis of uncommon events, share lessons within and across organizations, and improve patient safety culture.1 In our August 26, 2021 newsletter, the main article, Pump Up the Volume: Tips for Increasing Error Reporting and Decreasing Patient Harm, focused on how to optimize reporting and the capacity for learning about the human, technical, organizational, and environmental factors that determine the safety of the system as a whole. We have confidence that many of our readers are successfully “pumping up the volume” of reporting errors and close calls within their organizations, but we are concerned that some may be relying too heavily on error reporting as the only source of safety event data. Furthermore, we are concerned that some may use error report volume to calculate an error “rate” and then use this value as their only or primary measure of safety. Organizations must be mindful of the limitations of using reporting systems to generate error rates or less than impactful charts and graphs. Instead, the focus should be on individual error reports that reveal important actionable system issues, as well as aggregating and prioritizing reported events and investigating them thoroughly so meaningful system changes can be implemented and measured.
Suboptimal outcomes from internal reporting systems. Developing internal systems for reporting and tracking errors within a healthcare organization is the first step toward medication safety and promoting a Just Culture, but it is not enough. In the absence of effective error investigation and analysis, error reporting systems provide little insight into the safety of the medication-use system. Further, if an organization’s resources limit its ability to analyze and respond to large volumes of reports, staff attitudes toward reporting can darken, believing that their efforts and concerns are being ignored. We have seen all too often that an organization’s investigation and analysis of errors are superficial. As a result, meaningful change occurs infrequently, and outcomes often result in increased education rather than high-leverage system changes.1 Without contributing to external reporting systems, large-scale tracking and trend analysis are also lacking.2
Inappropriate use of error rates. Both ISMP and the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) recommend that due to differences in culture, definition, patient populations, and the types of reporting and detection systems, medication error rates should not be used to compare one organization to another. NCC MERP states, and ISMP fully agrees, that there is no acceptable incidence rate for medication errors and that the number of error reports is less important than the quality of the information collected, the organization’s analysis of the information, and its actions to improve the system to prevent harm to patients.
Large variations in the types of errors reported and what constitutes the threshold to report also exist. Practitioners are more likely to report an event based on the severity of the event or if the event occurred closer to or reached the patient. In addition, some provider types report adverse events regularly (e.g., nurses, pharmacists) while others report less frequently (e.g., physicians). The impact of all these variables on error reporting validates that error rates cannot be used to measure safety over time and should not be used for comparison between health systems, hospitals, and healthcare practitioners.
Safe Practice Recommendations: While “pumping up the volume” of reporting is an admirable goal, the ultimate measure of success for error-reporting programs is not the number of reports received but rather the learning that occurs, actions taken to improve the safety system, and the amount of patient harm prevented as a result of system changes prompted by the reports. When analyzing error data, consider the following:
Utilize error reporting as a barometer of safety culture. Organizations should utilize the quantity of errors reported in conjunction with surveys of the hospital’s safety culture to gauge the psychological safety individuals perceive in regard to error reporting. Rather than simply setting a goal of the number of events reported, take steps to strengthen the organization’s culture of safety. Then, measure changes in the number of self-reports and anonymous reports, as increases in self-reports may indicate that staff feel safe sharing experiences that have happened to them to avoid reoccurrence or the potential of the error actually reaching a patient. Anonymous reports might indicate the opposite and can be a barrier to understanding root causes since communication with the reporter for additional information is not possible. Similar to self-reports, an increase in reported close calls demonstrates the development of a learning culture, where individuals see value in sharing safety issues and trying to proactively solve them.
Avoid error rate comparison. Eliminate attempts to create medication error rates from reports for purposes of internal (e.g., unit-to-unit, employee-to-employee) and/or external comparisons (e.g., “benchmarking” error rates with other hospitals) to measure medication safety within the organization. Recognize that “error rates” are grossly inaccurate, because they are only based on voluntary reporting in an environment in which staff may be fearful to report an error, may not recognize that an error has occurred, or for a variety of reasons may choose not to report. If error reports are “counted,” educate practitioners that the goal is to elevate the “reporting” rate, not keep it low. In addition to sharing stories with staff, highlight the system-based problems that have been uncovered and the corresponding efforts and strategies employed to reduce the risk of errors and patient harm. Consider using ISMP’s conceptual model, the Key Elements of the Medication Use System™, to guide you through this process. While it may be difficult to measure risk avoidance and a reduction in patient harm, a reasonable alternative is measuring the number of system changes as a result of information shared through the error-reporting system.1
Share system changes and error report outcomes. When events are reported, be transparent with staff about the error that occurred without identifying individuals. Ask questions and talk freely with staff to learn if the circumstances related to the error still exist and help them to understand the risk. Discuss initial plans for process improvements and allow for feedback and questions so that they can all be part of the resolution of risk. Share this information across a variety of sources including with departmental staff (e.g., safety huddles, staff meetings, newsletters), cross-departmental colleagues (e.g., interprofessional huddles, committee meetings, interdepartmental newsletters), other hospitals/facilities within your health system, and leadership including the board of directors. If staff have input into the process and observe changes based on their reports and feedback, they will be more willing to take the time to report hazards and errors and participate in other performance/safety improvement projects. Sharing details of implemented system changes will also facilitate the spread of lessons learned and will help others to improve their systems.
Understand the significance of a trend. The reporting information may show a trend of similar reports, but it may not necessarily reveal what is truly happening within your organization. For example, if you identify an increase in reports of adverse reactions to a chemotherapy medication, further investigation as to why the increase in reporting is happening is needed. Was there a medication recall due to a manufacturing issue? A recent change in prescribing practices for oncology patients? Use of a new care plan or protocol? Has there been a change in pharmacy workflow during chemotherapy compounding, or a change in administration practices? Have the devices used in these processes changed? Ask staff for feedback including if there were any deviations in the process (e.g., products, equipment, technique) to gain insight into what might be contributing to this uptick in reports. Learn too about the identification of the adverse reaction. Were there changes in monitoring practices? While the safety report trend might be the initial call to action, collecting information from multiple sources gives meaning to the data and may point to system variability that needs attention.
Investigate rare events that could lead to harm. When prioritizing error reports for further follow-up, consider if the hazard or error is new and if it has caused or could cause harm. Does it require action by the US Food and Drug Administration (FDA), ISMP, the manufacturer, or standards organizations such as USP or The Joint Commission? Can patients, vendors, standards organizations, and regulators take specific actions to prevent or reduce the risk of similar errors or mitigate potential patient harm? For these instances, we cannot wait for a trend to occur, and the individual report is enough to escalate and prompt further action.
Detect errors through other means. To generate a more complete picture of the safety of the medication-use process, organizations must collect and analyze data beyond that gathered through voluntary error reporting. Whether they are included in your safety reporting system or not, pharmacy interventions can be a valuable data source, and organizations should not miss proactive opportunities and wait for an error to be reported to improve systems. Include errors detected and/or averted by automation (e.g., barcode scanning data, smart pump data, alerts generated in order entry and verification systems). Monitor medication-related triggers including the use of reversal agents (e.g., naloxone, flumazenil), abnormal laboratory results, procedure or treatment complications, and patient transfer to a higher level of care. While time consuming, you can learn a lot about process variation through observational studies of critical or complex parts of the process (e.g., pharmacy compounding, medication administration). Staff are often very willing to suggest at what points in the process they are feeling vulnerable; all you have to do is ask. Develop frontline staff safety round-table discussions for staff to discuss common issues seen in day-to-day practice to identify potential system improvements that may help prevent errors, and include these discussions in communications with leadership.
Proactively identify risks. Use tools such as the ISMP Medication Safety Self Assessments to assist your team in proactively identifying opportunities for reducing patient harm.
Learn from external reports. Reviewing and acting upon external reports such as those published in the ISMP Medication Safety Alert! is critical. Since these events are often rare, but can happen again elsewhere, they need to be shared and addressed proactively to protect patients. Not reviewing these reports and taking the recommended actions is one reason why we continue to see deaths from medication errors when we already know what to do to prevent them. Identify individuals, such as a medication safety officer (MSO) or members of an interdisciplinary medication safety committee, to regularly review external medication safety-related resources and include the following in the regular review:
-
ISMP Medication Safety Alert! publications, Action Agendas, and Targeted Medication Safety Best Practices for Hospitals (Best Practices)
-
The Joint Commission Sentinel Event Alert newsletters
-
US Food and Drug Administration (FDA) drug alerts and statements
-
National Alert Network (NAN) alerts
-
American Society of Health-System Pharmacists (ASHP) Connect list serve
Create a strategic plan. Use a model like the California Medication Error Reduction Plan (CA MERP) framework as a foundation for developing a strategic plan to advance the following error-reduction strategies:
-
Maintaining a robust medication error reporting system
-
Analyzing medication risks and errors with your interdisciplinary team
-
Carefully planning technology implementation
-
Using effective and timely measurable assessments to evaluate the impact of selective error-reduction strategies
-
Using a proactive approach to risk identification and analysis
-
Using external information to improve medication safety
-
Annually reviewing the strategic plan to reduce medication errors
Share errors with external entities. Organizations should collaborate with a Patient Safety Organization (PSO) to share adverse drug events or hazards and learn from other organizations. If more than one organization completes a root cause analysis (RCA) on the same issue and this information is aggregated, the outcome will have a larger impact that may otherwise have gone unnoticed.
Enhance event reporting systems. Work with vendors to modify internal reporting systems to meet end user needs including the ability to generate useful and customizable reports. Systems should be agile so that staff responsible for overseeing the reports can enter follow-up information at a later time pending a more time-intensive investigation. Developers of reporting systems should be flexible in their inclusion of new information technology, reflect advances in patient safety research, and be responsive to user feedback.3
References:
- Pham JC, Girard T, Pronovost PJ. What to do with healthcare incident reporting systems. J Public Health Res. 2013;2(3):e27.
- Smetzer J, Cohen MR. Medication error reporting systems. In: Cohen MR, ed. Medication Errors. 2nd ed. American Pharmacists Association; 2007;513-550.
- Nosek RA Jr, McMeekin J, Rake GW. Standardizing Medication Error Event Reporting in the U.S. Department of Defense. In: Henriksen K, Battles JB, Marks ES, Lewin DI, eds. Programs, Tools, and Products. Agency for Healthcare Research and Quality; 2005. Advances in Patient Safety: From Research to Implementation; vol 4.
Suggested citation:
Institute for Safe Medication Practices (ISMP). Pump up the volume: how to prioritize events and analyze error data. ISMP Medication Safety Alert! Acute Care. 2023;28(3):1-4.
