James A. Taylor, MD

The problem of medical errors in the United States has been well documented.1 There is evidence that pediatric patients may be at higher risk than are adult patients for certain types of errors.2 Ultimately, the only way to accurately assess whether pediatric patient safety is improved is by developing methodologies that will enable systematic counting of all medical errors. It is only through this technique that the effectiveness of interventions to improve safety can be adequately assessed. However, as a first step, it is crucial that data on at least a representative sample of medical errors occurring during the care of hospitalized children be collected so that the most common types and causes of these errors can be determined.

Many techniques have been used to collect data on medical errors including chart review, administrative data analysis, and malpractice claims analysis.35 Although each of these methodologies has advantages, each also has inherent biases in the types of errors that are detected. Direct observation of medical care is a powerful technique but has a number of limitations including cost.3 Voluntary or semivoluntary reporting systems have the potential to capture complete and representative information on errors, particularly near‐miss events. Voluntary reporting systems have been a highly successful method for understanding safety issues in other industries.6 In medicine, incident reports traditionally have been used as the main system for collecting data on a number of types of adverse events including medical errors.7 However, incident reports have been of limited use in understanding patient safety issues; only a small fraction of the errors made are reported, and certain types of errors are much more likely to be reported than others.4, 810 Medical professionals underreporting their own errors or those of their colleagues in incident reports may reflect fears that discovery of these errors will lead to embarrassment, job sanctions, or malpractice claims.1012

Cognizant of the tendency of professionals to underreport their errors, the aviation industry implemented a confidential reporting system for near‐miss events, the Aviation Safety Reporting System, in 1976.1 With this system, airline pilots file reports of near‐misses to a third party rather than to their employer, and the contents of the reports are kept confidential. Databases of the reports are anonymous. The implementation of the Aviation Safety Reporting System led to a substantial increase in reporting; analysis of the reports of near‐miss events has helped to significantly improve aviation safety in the past quarter century.1, 6 Based on the aviation experience, anonymous medical error reporting systems using either paper or Web‐based data entry have been implemented in adult intensive care units, neonatal intensive care units, and academic medical centers and for reporting specific types of errors.1318 There are limited data on whether these systems improve reporting of medical errors compared with use of the more traditional incident reporting systems already in place in virtually all hospitals.

We developed an online confidential and anonymous system for reporting medical errors in pediatric patients. For a 3‐month period this system replaced incident reports as the method by which medical errors were reported on 2 units in a large urban children's hospital. Data collected via the anonymous reporting system were compared with data in incident reports filed in the same 2 units during analogous 3‐month periods in the preceding 4 years. Prior to the study we postulated that substantially more medical errors would be reported through the anonymous system than through the incident reports and that information would be collected on a wider range of problems. It was hypothesized that reporting of near‐miss events would be particularly increased with the anonymous system.

METHODS

This study was conducted at Children's Hospital and Regional Medical Center (CHRMC), Seattle, Washington. CHRMC is both a community hospital serving pediatric patients and a tertiary‐care regional referral center. Two inpatient units, the infant intensive care unit (IICU) and the medical unit, participated in the project. The IICU provides care to critically ill neonates and infants up to 6 months of age; most patients admitted to the unit are premature newborns or newborns with congenital abnormalities. The medical unit is the major service for inpatient pediatric patients with nonsurgical problems. There 2 units were selected for the study because of a wide range of clinical problems, varying intensities of care and because of the clinical leadership's interest in patient safety issues.

Traditionally, medical errors at CHRMC have been documented through the use of a standard incident report system. However, during the 3‐month study period, from mid‐February through mid‐May 2003, physicians and nurses in the 2 study units were asked to report all medical errors using an electronic, anonymous reporting system that was installed on virtually all the computer workstations in the 2 units. Although all physicians and nurses were asked to use the anonymous system instead of completing incident reports, a physician or nurse who did not wish to participate in the research study could complete a standard incident report form as was consistent with hospital policy. Thus, medical errors were only reported once, either through the anonymous system for study participants or on incident reports for those who did not wish to participate in the project.

Before and during the data collection period, a member of the research team met with physicians on duty in the study units, including residents, fellows, and attending physicians, to explain the study procedures. Clinical nurse specialists in the study units provided the nursing staff with ongoing training based on a curriculum prepared specifically for the project. Topics covered in the training of both nurses and physicians included accessing the system, examples of medical errors, the importance of reporting errors, including near‐misses, and types of feedback provided. The anonymous nature of the reports was stressed, and the review procedures were explained.

During the study, nurses and physicians accessed the report form by clicking on an icon on a workstation desktop. The reporter was asked to provide the date and time when and the unit on which the event occurred. After filling in this information, the 2 dialog boxes on the form had to be completed. On the first, the reporter was asked to describe the event and on the second to report the outcome, if known, of the patient involved. All information on the form was completed using free text; there were no pull‐down menus or radio buttons. This was done to encourage more complete narratives and to be as inclusive as possible when asking nurses and physicians to report. Prior to the study, it was believed that asking potential reporters to classify whether events were errors or to classify them by type or other characterizations might keep nurses and physicians from reporting events that did not fit into a particular category and that a forced entry format would tend to reinforce current biases about errors rather than maximize the amount of new information gathered. Finally, to preserve anonymity, reporters were not asked to give any information about themselves, including profession (nurse or physician). However, they could provide their own names if they wanted feedback on the event, with the obvious loss of anonymity. Once the form was completed, the physician or nurse clicked the submit button to transmit the report to the research team.

A member of the research team reviewed every anonymous report within 48 hours of submission. If the event described was considered a medical error with the potential for serious patient injury, the investigator contacted a member of the clinical leadership of the unit (consisting of a medical director, one or more head nurses, and clinical nurse specialists) about the report. Every month members of the clinical leadership also received batched copies of all reports from their unit. Otherwise, neither the clinical nor the administrative leadership had access to the reports.

Each of the study's 3 pediatrician investigators (J.T., D.B., and E.K.) independently reviewed every report. First, the reviewer determined whether the event described constituted a medical error based on the definition provided by the Institute of Medicine.1 Events were further categorized by severity, occurrence to patient, and type. A medical error was considered serious if it resulted in or had the potential to result in permanent patient injury or death, moderately serious if it resulted in or had the potential to result in temporary physical or emotional injury, or trivial if it was unlikely to result in injury or change in treatment plan. Each error was further classified by whether it actually occurredeither as having actually happened to a patient or as being a near‐miss, an error detected before reaching the patient.

Because there is, to our knowledge, no standardized taxonomy for categorizing types of medical errors that occur in inpatient pediatric patients, a classification system was developed by the University of Washington Developmental Center for Evaluation and Research in Pediatric Patient Safety. (The developmental center and its organizational structure have been previously described).10 A preliminary classification system was patterned after the schema proposed by Leape et al. and adapted for use in pediatrics.19 After reviewing a series of incident reports for another project, the developers of this classification system for types of errors further refined it. The final taxonomy had 8 main types of medical errors, most with subtypes. The schema used for classifying types of errors in this study is shown in Table 1. Although the reviewers found frequent overlap, they determined the primary type of error for events described in each report based on this classification system. Final categorization of the errors, including severity, occurrence to patient, and type, was based on agreement by at least 2 of the 3 reviewers. In instances in which there was not sufficient agreement for categorization, the 3 reviewers reached a consensus after discussion.

For comparison, an identical review was conducted of incident reports completed in the 2 study units during the same months (mid‐February through mid‐May) in the years 1999‐2002. By including data from several previous years for comparison, the potential problem of selecting a period that was an outlier (in which one or more unusual factors led to increased or decreased reporting) was avoided. We selected the years 1999‐2002 because this was a period of increasing interest in better understanding medical errors at CHRMC. During this period, physicians and staff were encouraged to report medical errors, including near‐miss events, on incident reports. As with the anonymous electronic submissions, each investigator independently reviewed all the selected incident reports, with final classification based on the same schema used for the anonymous reports.

Comparison of the 2 reporting systems was complicated by the hospitalwide quality improvement program to increase the accuracy of labeling laboratory specimens that was ongoing during 1999‐2002. As part of this program, the hospital staff was encouraged to use the incident report system to document unlabeled or mismatched laboratory specimens and patients without proper identification from whom a laboratory specimen was to be obtained (eg, missing a hospital identification bracelet). Laboratory personnel completed most of these incident reports. In a previous review of incident report data from CHRMC, we found that 35% of medical errors reported were related to improper labeling of laboratory specimens (unpublished data). Although reporting these events may have been helpful for monitoring progress in quality improvement, many of the events described were extremely trivial in nature. Inclusion of this one specific type of event so skewed the overall number of medical errors reported that meaningful analysis of the types, relative frequencies, and reporting of errors was difficult. Based on this experience, we considered excluding this type of event from the analysis in the current study if it constituted a significant proportion of the medical errors conveyed in incident reports. Descriptions of mislabeled lab specimens or patients without identification bracelets constituted 33.8% of all incident reports from the 2 study units; no such events were described in submissions through the anonymous reporting system.

To compare the electronic anonymous and incident‐report error reporting systems, first the number of errors reported with each system was divided by the total number of patient‐days during which data were collected in the 2 units. Rates are expressed as the number of errors per 100 patient‐days. Rate ratios (RRs) with 95% confidence intervals (95% CIs) were calculated to compare the error reporting rates of the 2 reporting systems. Poisson regression was used to assess significance; a rate ratio whose 95% CI did not include 1.0 was considered statistically significant. Initial comparisons included all reports made through both systems. For subsequent comparisons, reports pertaining to mislabeled lab specimens were excluded. Error reporting rates were compared between the 2 reporting systems overall and by unit (medical unit and IICU), type, severity, and near‐miss status. In addition, to evaluate the possibility that secular trends in reporting medical errors were responsible for any observed overall differences, error reporting rates determined with the anonymous system were compared separately with incident report error rates in 1999, 2000, 2001, and 2002. Differences in the relative frequency of reporting different types of errors with the 2 systems were assessed with chi‐square tests. Kappa statistics were computed to assess the interobserver reliability of the 3 reviewers in classifying the events in the incident and anonymous reports as medical errors.

The study was approved by the Institutional Review Board of Children's Hospital and Regional Medical Center.

RESULTS

During the 3‐month study period, 146 reports were completed using the anonymous reporting system, 131 of which were classified as medical errors (89.7%). Ninety‐five errors were reported from the medical unit, and 36 were reported from the IICU. The kappa statistic for interobserver agreement in categorizing the anonymous reports as medical errors was .526. There were a total of 5420 patient‐days in the 2 units (medical service and IICU); thus, the rate of reporting medical errors via the anonymous system was 2.41/100 patient‐days (95% CI 2.02, 2.86). As shown in Table 2, the rate of errors reported in the IICU was higher than that in the medical unit. In addition to the errors reported via the anonymous system during the study period, 25 errors were reported using incident reports. Thus, the rate of reporting errors using both systems was 2.87.

A total of 633 incident reports were completed in the 2 study units during the analogous 3‐month periods in 1999‐2002, 538 of which were categorized as medical errors (85.0%). When all reports were considered, the rate of medical errors reported via the incident report system was 2.40/100 patient‐days (95% CI 2.21, 2.61). However, 17.3% of all errors reported in 1999, 37.2% of those reported in 2000, 40.2% of those in 2001, and 39.8% of those in 2002 pertained to mislabeled laboratory specimens. After excluding these reports, the overall rate of medical error reporting during 1999‐2002, calculated using incident report data, was 1.56/100‐patient days (95% CI 1.40, 1.73). The kappa statistic for interobserver agreement in classifying incident reports as medical errors was .615. Rates of error reporting in the medical unit and IICU are shown in Table 2.

After excluding reports dealing with mislabeled laboratory specimens, the error reporting rate was significantly higher using the anonymous system than using incident reports (RR 1.54, 95% CI 1.26, 1.90). The rate of reporting errors with the anonymous system was higher than those for reporting via incident reports in 1999, 2001, and 2002; there was no significant difference in reporting rates when the data collected with the anonymous system were compared with the data on errors reported via incident reports in 2000 (RR 1.26, 95% CI 0.97, 1.67; Table 2).

Much of the increased rate of reporting via the anonymous system came from the medical unit. The medical unit had an overall RR for anonymous reporting compared with incidence report submission of 1.77 (95% CI 1.31, 2.14); the rate of reporting via the anonymous system was significantly higher than via incident reports for each of the years 1999‐2002. Conversely, the rate of reporting observed in the IICU was not significantly increased (RR 1.33, 95% CI 0.89, 1.95, P = .07).

The types of errors reported with the 2 systems are summarized in Table 3. Although the overall distribution was only marginally different between the 2 systems (P = .054), a higher proportion of the errors reported via the anonymous system were medication errors (P = .019), whereas a higher percentage of errors reported with incident reports dealt with equipment failures (P = .033). The rate of reporting medication errors with the anonymous system (1.57 reports/100 patient‐days) was significantly higher than that via incident reports (0.83 reports/100 patient days, RR 1.90, 95% CI 1.44, 2.47). When compared with the individual years for which incident report data were available, the reporting rate for medication errors was significantly higher via the anonymous system than with incident reports for each of the years 1999‐2002.

The severity of medical errors reported with the 2 systems is shown in Table 4. As can be seen, errors reported via the anonymous system and in incident reports had a similar distribution of severity, with almost 80% of medical errors classified as moderately serious. The rate of reporting serious medical errors was 0.37/100 patient‐days with the anonymous system and 0.23/100 patient‐days via incident reports (RR 1.61, 95% CI 0.91, 2.76).

With the anonymous system, 25.2% of reported medical errors were near‐misses compared with 12.6% of the errors reported with the incident report system (P = .001). The rate of reporting near‐miss medical errors was 3‐fold higher with the anonymous system relative to reporting via incident reports (RR 3.10, 95% CI 1.91, 4.98) and was significantly higher than in each of the years data on incident reports were collected and in each of the 2 units. The reporting of errors that reached the patient was also significantly more frequent with the anonymous system than via incident reports; however, this increase was less pronounced (RR 1.32, 95% CI 1.05, 1.67). Among the 33 near‐miss events reported via the anonymous system were 10 medical errors categorized as serious. Six of these were related to medications, including two 10‐fold overdoses of morphine. Overall, the rate of reporting near‐miss medication errors was significantly higher with the anonymous system than with incident reports (RR 3.10, 95% CI 1.81, 5.24).

DISCUSSION

The results of this study suggest that implementation of an anonymous system was associated with a modest increase in the reporting of medical errors during the care of hospitalized children compared with reporting via a traditional incident report system. After excluding reports of mislabeled laboratory specimens, reported as part of a specific quality improvement project, the rate of errors reported with the anonymous system was approximately 54% higher than that using incident reports. The most striking upsurge in reporting observed with the anonymous system was the 3‐fold increase in reporting of near‐miss medical errors.

Because of different types of patients, lack of denominator data, different durations of observation, and, presumably, different inherent rates of errors, it is difficult to compare different anonymous reporting systems for medical errors. In one of the few studies dealing with pediatric patients, Suresh et al, evaluated a Web‐based anonymous reporting system in 54 neonatal intensive care units (NICUs).16 Over a 27‐month period, 1230 reports were completed via the system, for an average of slightly less than 1 report per NICU per month. This is substantially lower than the 12 errors per month reported from the IICU in our study using the anonymous system. In a study of a Web‐based anonymous system used by 18 ICUs in 11 hospitals, 854 reports were filed during a 12‐month period. The average rate of reporting ranged from 4.3 to 7.5 reports per ICU per month, with an overall mean of 6.5 reports per hospital per month.1415 However, unlike in our study, in which the anonymous system temporarily supplanted incident reports, only 2 of the 11 hospitals discontinued incident reporting.14 A national Web‐based system has been established for reporting medication errors. During a 2‐year period beginning in 1999, 154,816 medication errors were reported from 403 hospitals, for an average of 16 reports per hospital per month.18 This is less than the 28 medication errors reported per month with our anonymous system.

Anonymous systems based at a single institution have been associated with higher rates of reporting. In one study, approximately 68 events were reported per month during the first 16 weeks after full implementation of a hospitalwide anonymous system, compared with the average of 44 errors reported monthly in our project.17 In the study perhaps most comparable to ours, Osmon et al. reported on the use of an anonymously completed paper form used to report medical errors in an adult ICU.13 Patient safety advocates extensively described and promoted the reporting system prior to its use and while it was implemented. During the 6‐month study period, 8.93 medical events/100 patient‐days were reported with the system. This rate of reporting was 10‐fold higher than that reported via the standard reporting system used at that hospital.

In addition to rate of reporting medical errors, our study was designed to compare some aspects of the content of anonymous and incident reports. No statistically significant difference was found in the severity of the events reported; the rate of reporting serious medical errors was comparable between the 2 systems. This might suggest serious errors are the most likely to be reported regardless of the system used. However, given the modest number of serious events reported with either the anonymous or the incident report system (20 and 51, respectively), the power to detect a significant difference in rates was limited. Conversely, implementation of the anonymous system was associated with increased reporting of near‐miss events of all types and was a particularly useful mechanism for reporting near‐miss medication errors. Because near‐miss events may not be detected by other methods for identifying medication errors such as chart review or search for specific triggers, the use of an anonymous system may be an important tool in a multifaceted effort to improve medication safety. Perhaps the best use of an online anonymous system would be to provide a mechanism for rapid reporting of near‐miss errors, whereas other systems, such as incident reports, could be used to report errors that reach the patient.

We were surprised that although the reporting of medical errors was increased on the medical unit with the implementation of the anonymous system, there was no significant change in overall reporting in the IICU. This was possibly because reporting via incident reports was already more complete in the IICU, so that a small increase with the anonymous system was less likely to be detected However, it is equally plausible that because of the severity of illness of the patients in the IICU, physicians and staff in this unit had a perception that they did not have enough free time to report all errors. Finally, it is possible that the staff and/or clinical leadership in the medical unit was more enthusiastic about the anonymous system. Regardless, this result suggests that despite training on reporting, provision of an easy‐to‐use system, and the guarantee of anonymity, significant barriers to reporting medical errors remain.

The Kappa statistic of .526 for level of agreement between reviewers in categorizing events described with the anonymous system as medical errors indicates only a good level of agreement.20 This lack of agreement may be in part a result of the limited amount of information provided in some of the narrative reports of events. Because anonymous reports did not include names of patients or providers, it was impossible to review medical records or other information to gain additional information about the events described. However, as pointed out by others, determination of when a medical error has occurred, although seemingly simple, is frequently much less clear when reviewing actual events.21

The findings in our study should be interpreted cautiously. Because of the need for a unified system to record events across the entire hospital, anonymous reports supplanted incident reports in the 2 study units for only a 3‐month period; it is impossible to predict the long‐term trends in reporting with this system. We selected the winterspring period for the study because it is a busy time of year for children's hospitals. Rates of reporting and medical errors may change dramatically during other times of the year, particularly in a teaching hospital. An underlying assumption of our comparisons between the 2 reporting systems was that the actual rate of medical errors was unchanged throughout the period and that the differences observed were a result of more complete reporting with the anonymous system. The increased rate of reporting of medical errors found with the anonymous reporting system might have been influenced by the training given the medical personnel. It is also possible that the increased reporting rates with the anonymous system occurred because of increased publicity, both in the press and in the hospital, about medical errors and patient safety, in general. However, because there was no definite secular trend in reporting observed during the years 1999‐2002, it is unlikely that this explains our findings. Finally, it is impossible to measure the relative impact of the increased ease of reporting with the online system versus the anonymity provided.

Although the anonymous system was associated with a 54% increase in rate of reporting, it is clear that the vast majority of medical errors were not reported. If the estimates that incident reports capture 1%‐10% of errors are accurate,8, 9 the increase in reporting that we observed with the anonymous system would indicate that 1.5%15% of errors were reported. The impressive 10‐fold increase in reporting observed by Osmon et al. in their study of an anonymous system was partly a result of the very low rate of reporting with their traditional system (approximately .67 reports of medical errors/100 adult ICU patient‐days).13 A common feature of studies of anonymous systems with higher rates of reporting medical errors is the continuing presence of on‐site patient safety investigators and advocates.13, 17 Rather than the particulars of the reporting system used, this on‐site presence and advocacy may be the most important element in increasing voluntary reporting of medical errors. In our study it is likely that some of the increase in reporting observed with the anonymous system was related to publicity about the system and ongoing promotion of the importance of reporting errors by the research team.

Since completion of the study, CHRMC has been using incident reports as the main tool for collecting data on medical errors in all units. However, based on our experiences, a new reporting tool, called e‐feedback, has been instituted. The goal of this system is to allow physicians and staff members to quickly report events that may be indicative of systems problems in the delivery of care. The reports are reviewed by designated multidisciplinary teams in various units throughout the hospital so that changes can be implemented, if needed.

CONCLUSIONS

Although there was a modest increase in the number of reports, the results of this study indicate that the implementation of an anonymous online reporting system (with training on the use of the system) was not a panacea for the problem of underreporting of medical error. Use of a system such as we have described may be an effective tool for increasing the reporting of near‐miss events., However, our results suggest that methodologies in addition to voluntary or semivoluntary reporting systems are needed to more fully collect information on medical errors.

The problem of medical errors in the United States has been well documented.1 There is evidence that pediatric patients may be at higher risk than are adult patients for certain types of errors.2 Ultimately, the only way to accurately assess whether pediatric patient safety is improved is by developing methodologies that will enable systematic counting of all medical errors. It is only through this technique that the effectiveness of interventions to improve safety can be adequately assessed. However, as a first step, it is crucial that data on at least a representative sample of medical errors occurring during the care of hospitalized children be collected so that the most common types and causes of these errors can be determined.

Many techniques have been used to collect data on medical errors including chart review, administrative data analysis, and malpractice claims analysis.35 Although each of these methodologies has advantages, each also has inherent biases in the types of errors that are detected. Direct observation of medical care is a powerful technique but has a number of limitations including cost.3 Voluntary or semivoluntary reporting systems have the potential to capture complete and representative information on errors, particularly near‐miss events. Voluntary reporting systems have been a highly successful method for understanding safety issues in other industries.6 In medicine, incident reports traditionally have been used as the main system for collecting data on a number of types of adverse events including medical errors.7 However, incident reports have been of limited use in understanding patient safety issues; only a small fraction of the errors made are reported, and certain types of errors are much more likely to be reported than others.4, 810 Medical professionals underreporting their own errors or those of their colleagues in incident reports may reflect fears that discovery of these errors will lead to embarrassment, job sanctions, or malpractice claims.1012

Cognizant of the tendency of professionals to underreport their errors, the aviation industry implemented a confidential reporting system for near‐miss events, the Aviation Safety Reporting System, in 1976.1 With this system, airline pilots file reports of near‐misses to a third party rather than to their employer, and the contents of the reports are kept confidential. Databases of the reports are anonymous. The implementation of the Aviation Safety Reporting System led to a substantial increase in reporting; analysis of the reports of near‐miss events has helped to significantly improve aviation safety in the past quarter century.1, 6 Based on the aviation experience, anonymous medical error reporting systems using either paper or Web‐based data entry have been implemented in adult intensive care units, neonatal intensive care units, and academic medical centers and for reporting specific types of errors.1318 There are limited data on whether these systems improve reporting of medical errors compared with use of the more traditional incident reporting systems already in place in virtually all hospitals.

We developed an online confidential and anonymous system for reporting medical errors in pediatric patients. For a 3‐month period this system replaced incident reports as the method by which medical errors were reported on 2 units in a large urban children's hospital. Data collected via the anonymous reporting system were compared with data in incident reports filed in the same 2 units during analogous 3‐month periods in the preceding 4 years. Prior to the study we postulated that substantially more medical errors would be reported through the anonymous system than through the incident reports and that information would be collected on a wider range of problems. It was hypothesized that reporting of near‐miss events would be particularly increased with the anonymous system.

METHODS

This study was conducted at Children's Hospital and Regional Medical Center (CHRMC), Seattle, Washington. CHRMC is both a community hospital serving pediatric patients and a tertiary‐care regional referral center. Two inpatient units, the infant intensive care unit (IICU) and the medical unit, participated in the project. The IICU provides care to critically ill neonates and infants up to 6 months of age; most patients admitted to the unit are premature newborns or newborns with congenital abnormalities. The medical unit is the major service for inpatient pediatric patients with nonsurgical problems. There 2 units were selected for the study because of a wide range of clinical problems, varying intensities of care and because of the clinical leadership's interest in patient safety issues.

Traditionally, medical errors at CHRMC have been documented through the use of a standard incident report system. However, during the 3‐month study period, from mid‐February through mid‐May 2003, physicians and nurses in the 2 study units were asked to report all medical errors using an electronic, anonymous reporting system that was installed on virtually all the computer workstations in the 2 units. Although all physicians and nurses were asked to use the anonymous system instead of completing incident reports, a physician or nurse who did not wish to participate in the research study could complete a standard incident report form as was consistent with hospital policy. Thus, medical errors were only reported once, either through the anonymous system for study participants or on incident reports for those who did not wish to participate in the project.

Before and during the data collection period, a member of the research team met with physicians on duty in the study units, including residents, fellows, and attending physicians, to explain the study procedures. Clinical nurse specialists in the study units provided the nursing staff with ongoing training based on a curriculum prepared specifically for the project. Topics covered in the training of both nurses and physicians included accessing the system, examples of medical errors, the importance of reporting errors, including near‐misses, and types of feedback provided. The anonymous nature of the reports was stressed, and the review procedures were explained.

During the study, nurses and physicians accessed the report form by clicking on an icon on a workstation desktop. The reporter was asked to provide the date and time when and the unit on which the event occurred. After filling in this information, the 2 dialog boxes on the form had to be completed. On the first, the reporter was asked to describe the event and on the second to report the outcome, if known, of the patient involved. All information on the form was completed using free text; there were no pull‐down menus or radio buttons. This was done to encourage more complete narratives and to be as inclusive as possible when asking nurses and physicians to report. Prior to the study, it was believed that asking potential reporters to classify whether events were errors or to classify them by type or other characterizations might keep nurses and physicians from reporting events that did not fit into a particular category and that a forced entry format would tend to reinforce current biases about errors rather than maximize the amount of new information gathered. Finally, to preserve anonymity, reporters were not asked to give any information about themselves, including profession (nurse or physician). However, they could provide their own names if they wanted feedback on the event, with the obvious loss of anonymity. Once the form was completed, the physician or nurse clicked the submit button to transmit the report to the research team.

A member of the research team reviewed every anonymous report within 48 hours of submission. If the event described was considered a medical error with the potential for serious patient injury, the investigator contacted a member of the clinical leadership of the unit (consisting of a medical director, one or more head nurses, and clinical nurse specialists) about the report. Every month members of the clinical leadership also received batched copies of all reports from their unit. Otherwise, neither the clinical nor the administrative leadership had access to the reports.

Each of the study's 3 pediatrician investigators (J.T., D.B., and E.K.) independently reviewed every report. First, the reviewer determined whether the event described constituted a medical error based on the definition provided by the Institute of Medicine.1 Events were further categorized by severity, occurrence to patient, and type. A medical error was considered serious if it resulted in or had the potential to result in permanent patient injury or death, moderately serious if it resulted in or had the potential to result in temporary physical or emotional injury, or trivial if it was unlikely to result in injury or change in treatment plan. Each error was further classified by whether it actually occurredeither as having actually happened to a patient or as being a near‐miss, an error detected before reaching the patient.

Because there is, to our knowledge, no standardized taxonomy for categorizing types of medical errors that occur in inpatient pediatric patients, a classification system was developed by the University of Washington Developmental Center for Evaluation and Research in Pediatric Patient Safety. (The developmental center and its organizational structure have been previously described).10 A preliminary classification system was patterned after the schema proposed by Leape et al. and adapted for use in pediatrics.19 After reviewing a series of incident reports for another project, the developers of this classification system for types of errors further refined it. The final taxonomy had 8 main types of medical errors, most with subtypes. The schema used for classifying types of errors in this study is shown in Table 1. Although the reviewers found frequent overlap, they determined the primary type of error for events described in each report based on this classification system. Final categorization of the errors, including severity, occurrence to patient, and type, was based on agreement by at least 2 of the 3 reviewers. In instances in which there was not sufficient agreement for categorization, the 3 reviewers reached a consensus after discussion.

For comparison, an identical review was conducted of incident reports completed in the 2 study units during the same months (mid‐February through mid‐May) in the years 1999‐2002. By including data from several previous years for comparison, the potential problem of selecting a period that was an outlier (in which one or more unusual factors led to increased or decreased reporting) was avoided. We selected the years 1999‐2002 because this was a period of increasing interest in better understanding medical errors at CHRMC. During this period, physicians and staff were encouraged to report medical errors, including near‐miss events, on incident reports. As with the anonymous electronic submissions, each investigator independently reviewed all the selected incident reports, with final classification based on the same schema used for the anonymous reports.

Comparison of the 2 reporting systems was complicated by the hospitalwide quality improvement program to increase the accuracy of labeling laboratory specimens that was ongoing during 1999‐2002. As part of this program, the hospital staff was encouraged to use the incident report system to document unlabeled or mismatched laboratory specimens and patients without proper identification from whom a laboratory specimen was to be obtained (eg, missing a hospital identification bracelet). Laboratory personnel completed most of these incident reports. In a previous review of incident report data from CHRMC, we found that 35% of medical errors reported were related to improper labeling of laboratory specimens (unpublished data). Although reporting these events may have been helpful for monitoring progress in quality improvement, many of the events described were extremely trivial in nature. Inclusion of this one specific type of event so skewed the overall number of medical errors reported that meaningful analysis of the types, relative frequencies, and reporting of errors was difficult. Based on this experience, we considered excluding this type of event from the analysis in the current study if it constituted a significant proportion of the medical errors conveyed in incident reports. Descriptions of mislabeled lab specimens or patients without identification bracelets constituted 33.8% of all incident reports from the 2 study units; no such events were described in submissions through the anonymous reporting system.

To compare the electronic anonymous and incident‐report error reporting systems, first the number of errors reported with each system was divided by the total number of patient‐days during which data were collected in the 2 units. Rates are expressed as the number of errors per 100 patient‐days. Rate ratios (RRs) with 95% confidence intervals (95% CIs) were calculated to compare the error reporting rates of the 2 reporting systems. Poisson regression was used to assess significance; a rate ratio whose 95% CI did not include 1.0 was considered statistically significant. Initial comparisons included all reports made through both systems. For subsequent comparisons, reports pertaining to mislabeled lab specimens were excluded. Error reporting rates were compared between the 2 reporting systems overall and by unit (medical unit and IICU), type, severity, and near‐miss status. In addition, to evaluate the possibility that secular trends in reporting medical errors were responsible for any observed overall differences, error reporting rates determined with the anonymous system were compared separately with incident report error rates in 1999, 2000, 2001, and 2002. Differences in the relative frequency of reporting different types of errors with the 2 systems were assessed with chi‐square tests. Kappa statistics were computed to assess the interobserver reliability of the 3 reviewers in classifying the events in the incident and anonymous reports as medical errors.

The study was approved by the Institutional Review Board of Children's Hospital and Regional Medical Center.

RESULTS

During the 3‐month study period, 146 reports were completed using the anonymous reporting system, 131 of which were classified as medical errors (89.7%). Ninety‐five errors were reported from the medical unit, and 36 were reported from the IICU. The kappa statistic for interobserver agreement in categorizing the anonymous reports as medical errors was .526. There were a total of 5420 patient‐days in the 2 units (medical service and IICU); thus, the rate of reporting medical errors via the anonymous system was 2.41/100 patient‐days (95% CI 2.02, 2.86). As shown in Table 2, the rate of errors reported in the IICU was higher than that in the medical unit. In addition to the errors reported via the anonymous system during the study period, 25 errors were reported using incident reports. Thus, the rate of reporting errors using both systems was 2.87.

A total of 633 incident reports were completed in the 2 study units during the analogous 3‐month periods in 1999‐2002, 538 of which were categorized as medical errors (85.0%). When all reports were considered, the rate of medical errors reported via the incident report system was 2.40/100 patient‐days (95% CI 2.21, 2.61). However, 17.3% of all errors reported in 1999, 37.2% of those reported in 2000, 40.2% of those in 2001, and 39.8% of those in 2002 pertained to mislabeled laboratory specimens. After excluding these reports, the overall rate of medical error reporting during 1999‐2002, calculated using incident report data, was 1.56/100‐patient days (95% CI 1.40, 1.73). The kappa statistic for interobserver agreement in classifying incident reports as medical errors was .615. Rates of error reporting in the medical unit and IICU are shown in Table 2.

After excluding reports dealing with mislabeled laboratory specimens, the error reporting rate was significantly higher using the anonymous system than using incident reports (RR 1.54, 95% CI 1.26, 1.90). The rate of reporting errors with the anonymous system was higher than those for reporting via incident reports in 1999, 2001, and 2002; there was no significant difference in reporting rates when the data collected with the anonymous system were compared with the data on errors reported via incident reports in 2000 (RR 1.26, 95% CI 0.97, 1.67; Table 2).

Much of the increased rate of reporting via the anonymous system came from the medical unit. The medical unit had an overall RR for anonymous reporting compared with incidence report submission of 1.77 (95% CI 1.31, 2.14); the rate of reporting via the anonymous system was significantly higher than via incident reports for each of the years 1999‐2002. Conversely, the rate of reporting observed in the IICU was not significantly increased (RR 1.33, 95% CI 0.89, 1.95, P = .07).

The types of errors reported with the 2 systems are summarized in Table 3. Although the overall distribution was only marginally different between the 2 systems (P = .054), a higher proportion of the errors reported via the anonymous system were medication errors (P = .019), whereas a higher percentage of errors reported with incident reports dealt with equipment failures (P = .033). The rate of reporting medication errors with the anonymous system (1.57 reports/100 patient‐days) was significantly higher than that via incident reports (0.83 reports/100 patient days, RR 1.90, 95% CI 1.44, 2.47). When compared with the individual years for which incident report data were available, the reporting rate for medication errors was significantly higher via the anonymous system than with incident reports for each of the years 1999‐2002.

The severity of medical errors reported with the 2 systems is shown in Table 4. As can be seen, errors reported via the anonymous system and in incident reports had a similar distribution of severity, with almost 80% of medical errors classified as moderately serious. The rate of reporting serious medical errors was 0.37/100 patient‐days with the anonymous system and 0.23/100 patient‐days via incident reports (RR 1.61, 95% CI 0.91, 2.76).

With the anonymous system, 25.2% of reported medical errors were near‐misses compared with 12.6% of the errors reported with the incident report system (P = .001). The rate of reporting near‐miss medical errors was 3‐fold higher with the anonymous system relative to reporting via incident reports (RR 3.10, 95% CI 1.91, 4.98) and was significantly higher than in each of the years data on incident reports were collected and in each of the 2 units. The reporting of errors that reached the patient was also significantly more frequent with the anonymous system than via incident reports; however, this increase was less pronounced (RR 1.32, 95% CI 1.05, 1.67). Among the 33 near‐miss events reported via the anonymous system were 10 medical errors categorized as serious. Six of these were related to medications, including two 10‐fold overdoses of morphine. Overall, the rate of reporting near‐miss medication errors was significantly higher with the anonymous system than with incident reports (RR 3.10, 95% CI 1.81, 5.24).

DISCUSSION

The results of this study suggest that implementation of an anonymous system was associated with a modest increase in the reporting of medical errors during the care of hospitalized children compared with reporting via a traditional incident report system. After excluding reports of mislabeled laboratory specimens, reported as part of a specific quality improvement project, the rate of errors reported with the anonymous system was approximately 54% higher than that using incident reports. The most striking upsurge in reporting observed with the anonymous system was the 3‐fold increase in reporting of near‐miss medical errors.

Because of different types of patients, lack of denominator data, different durations of observation, and, presumably, different inherent rates of errors, it is difficult to compare different anonymous reporting systems for medical errors. In one of the few studies dealing with pediatric patients, Suresh et al, evaluated a Web‐based anonymous reporting system in 54 neonatal intensive care units (NICUs).16 Over a 27‐month period, 1230 reports were completed via the system, for an average of slightly less than 1 report per NICU per month. This is substantially lower than the 12 errors per month reported from the IICU in our study using the anonymous system. In a study of a Web‐based anonymous system used by 18 ICUs in 11 hospitals, 854 reports were filed during a 12‐month period. The average rate of reporting ranged from 4.3 to 7.5 reports per ICU per month, with an overall mean of 6.5 reports per hospital per month.1415 However, unlike in our study, in which the anonymous system temporarily supplanted incident reports, only 2 of the 11 hospitals discontinued incident reporting.14 A national Web‐based system has been established for reporting medication errors. During a 2‐year period beginning in 1999, 154,816 medication errors were reported from 403 hospitals, for an average of 16 reports per hospital per month.18 This is less than the 28 medication errors reported per month with our anonymous system.

Anonymous systems based at a single institution have been associated with higher rates of reporting. In one study, approximately 68 events were reported per month during the first 16 weeks after full implementation of a hospitalwide anonymous system, compared with the average of 44 errors reported monthly in our project.17 In the study perhaps most comparable to ours, Osmon et al. reported on the use of an anonymously completed paper form used to report medical errors in an adult ICU.13 Patient safety advocates extensively described and promoted the reporting system prior to its use and while it was implemented. During the 6‐month study period, 8.93 medical events/100 patient‐days were reported with the system. This rate of reporting was 10‐fold higher than that reported via the standard reporting system used at that hospital.

In addition to rate of reporting medical errors, our study was designed to compare some aspects of the content of anonymous and incident reports. No statistically significant difference was found in the severity of the events reported; the rate of reporting serious medical errors was comparable between the 2 systems. This might suggest serious errors are the most likely to be reported regardless of the system used. However, given the modest number of serious events reported with either the anonymous or the incident report system (20 and 51, respectively), the power to detect a significant difference in rates was limited. Conversely, implementation of the anonymous system was associated with increased reporting of near‐miss events of all types and was a particularly useful mechanism for reporting near‐miss medication errors. Because near‐miss events may not be detected by other methods for identifying medication errors such as chart review or search for specific triggers, the use of an anonymous system may be an important tool in a multifaceted effort to improve medication safety. Perhaps the best use of an online anonymous system would be to provide a mechanism for rapid reporting of near‐miss errors, whereas other systems, such as incident reports, could be used to report errors that reach the patient.

We were surprised that although the reporting of medical errors was increased on the medical unit with the implementation of the anonymous system, there was no significant change in overall reporting in the IICU. This was possibly because reporting via incident reports was already more complete in the IICU, so that a small increase with the anonymous system was less likely to be detected However, it is equally plausible that because of the severity of illness of the patients in the IICU, physicians and staff in this unit had a perception that they did not have enough free time to report all errors. Finally, it is possible that the staff and/or clinical leadership in the medical unit was more enthusiastic about the anonymous system. Regardless, this result suggests that despite training on reporting, provision of an easy‐to‐use system, and the guarantee of anonymity, significant barriers to reporting medical errors remain.

The Kappa statistic of .526 for level of agreement between reviewers in categorizing events described with the anonymous system as medical errors indicates only a good level of agreement.20 This lack of agreement may be in part a result of the limited amount of information provided in some of the narrative reports of events. Because anonymous reports did not include names of patients or providers, it was impossible to review medical records or other information to gain additional information about the events described. However, as pointed out by others, determination of when a medical error has occurred, although seemingly simple, is frequently much less clear when reviewing actual events.21

The findings in our study should be interpreted cautiously. Because of the need for a unified system to record events across the entire hospital, anonymous reports supplanted incident reports in the 2 study units for only a 3‐month period; it is impossible to predict the long‐term trends in reporting with this system. We selected the winterspring period for the study because it is a busy time of year for children's hospitals. Rates of reporting and medical errors may change dramatically during other times of the year, particularly in a teaching hospital. An underlying assumption of our comparisons between the 2 reporting systems was that the actual rate of medical errors was unchanged throughout the period and that the differences observed were a result of more complete reporting with the anonymous system. The increased rate of reporting of medical errors found with the anonymous reporting system might have been influenced by the training given the medical personnel. It is also possible that the increased reporting rates with the anonymous system occurred because of increased publicity, both in the press and in the hospital, about medical errors and patient safety, in general. However, because there was no definite secular trend in reporting observed during the years 1999‐2002, it is unlikely that this explains our findings. Finally, it is impossible to measure the relative impact of the increased ease of reporting with the online system versus the anonymity provided.

Although the anonymous system was associated with a 54% increase in rate of reporting, it is clear that the vast majority of medical errors were not reported. If the estimates that incident reports capture 1%‐10% of errors are accurate,8, 9 the increase in reporting that we observed with the anonymous system would indicate that 1.5%15% of errors were reported. The impressive 10‐fold increase in reporting observed by Osmon et al. in their study of an anonymous system was partly a result of the very low rate of reporting with their traditional system (approximately .67 reports of medical errors/100 adult ICU patient‐days).13 A common feature of studies of anonymous systems with higher rates of reporting medical errors is the continuing presence of on‐site patient safety investigators and advocates.13, 17 Rather than the particulars of the reporting system used, this on‐site presence and advocacy may be the most important element in increasing voluntary reporting of medical errors. In our study it is likely that some of the increase in reporting observed with the anonymous system was related to publicity about the system and ongoing promotion of the importance of reporting errors by the research team.

Since completion of the study, CHRMC has been using incident reports as the main tool for collecting data on medical errors in all units. However, based on our experiences, a new reporting tool, called e‐feedback, has been instituted. The goal of this system is to allow physicians and staff members to quickly report events that may be indicative of systems problems in the delivery of care. The reports are reviewed by designated multidisciplinary teams in various units throughout the hospital so that changes can be implemented, if needed.

CONCLUSIONS

Although there was a modest increase in the number of reports, the results of this study indicate that the implementation of an anonymous online reporting system (with training on the use of the system) was not a panacea for the problem of underreporting of medical error. Use of a system such as we have described may be an effective tool for increasing the reporting of near‐miss events., However, our results suggest that methodologies in addition to voluntary or semivoluntary reporting systems are needed to more fully collect information on medical errors.

The problem of medical errors in the United States has been well documented.1 There is evidence that pediatric patients may be at higher risk than are adult patients for certain types of errors.2 Ultimately, the only way to accurately assess whether pediatric patient safety is improved is by developing methodologies that will enable systematic counting of all medical errors. It is only through this technique that the effectiveness of interventions to improve safety can be adequately assessed. However, as a first step, it is crucial that data on at least a representative sample of medical errors occurring during the care of hospitalized children be collected so that the most common types and causes of these errors can be determined.

Many techniques have been used to collect data on medical errors including chart review, administrative data analysis, and malpractice claims analysis.35 Although each of these methodologies has advantages, each also has inherent biases in the types of errors that are detected. Direct observation of medical care is a powerful technique but has a number of limitations including cost.3 Voluntary or semivoluntary reporting systems have the potential to capture complete and representative information on errors, particularly near‐miss events. Voluntary reporting systems have been a highly successful method for understanding safety issues in other industries.6 In medicine, incident reports traditionally have been used as the main system for collecting data on a number of types of adverse events including medical errors.7 However, incident reports have been of limited use in understanding patient safety issues; only a small fraction of the errors made are reported, and certain types of errors are much more likely to be reported than others.4, 810 Medical professionals underreporting their own errors or those of their colleagues in incident reports may reflect fears that discovery of these errors will lead to embarrassment, job sanctions, or malpractice claims.1012

Cognizant of the tendency of professionals to underreport their errors, the aviation industry implemented a confidential reporting system for near‐miss events, the Aviation Safety Reporting System, in 1976.1 With this system, airline pilots file reports of near‐misses to a third party rather than to their employer, and the contents of the reports are kept confidential. Databases of the reports are anonymous. The implementation of the Aviation Safety Reporting System led to a substantial increase in reporting; analysis of the reports of near‐miss events has helped to significantly improve aviation safety in the past quarter century.1, 6 Based on the aviation experience, anonymous medical error reporting systems using either paper or Web‐based data entry have been implemented in adult intensive care units, neonatal intensive care units, and academic medical centers and for reporting specific types of errors.1318 There are limited data on whether these systems improve reporting of medical errors compared with use of the more traditional incident reporting systems already in place in virtually all hospitals.

We developed an online confidential and anonymous system for reporting medical errors in pediatric patients. For a 3‐month period this system replaced incident reports as the method by which medical errors were reported on 2 units in a large urban children's hospital. Data collected via the anonymous reporting system were compared with data in incident reports filed in the same 2 units during analogous 3‐month periods in the preceding 4 years. Prior to the study we postulated that substantially more medical errors would be reported through the anonymous system than through the incident reports and that information would be collected on a wider range of problems. It was hypothesized that reporting of near‐miss events would be particularly increased with the anonymous system.

METHODS

This study was conducted at Children's Hospital and Regional Medical Center (CHRMC), Seattle, Washington. CHRMC is both a community hospital serving pediatric patients and a tertiary‐care regional referral center. Two inpatient units, the infant intensive care unit (IICU) and the medical unit, participated in the project. The IICU provides care to critically ill neonates and infants up to 6 months of age; most patients admitted to the unit are premature newborns or newborns with congenital abnormalities. The medical unit is the major service for inpatient pediatric patients with nonsurgical problems. There 2 units were selected for the study because of a wide range of clinical problems, varying intensities of care and because of the clinical leadership's interest in patient safety issues.

Traditionally, medical errors at CHRMC have been documented through the use of a standard incident report system. However, during the 3‐month study period, from mid‐February through mid‐May 2003, physicians and nurses in the 2 study units were asked to report all medical errors using an electronic, anonymous reporting system that was installed on virtually all the computer workstations in the 2 units. Although all physicians and nurses were asked to use the anonymous system instead of completing incident reports, a physician or nurse who did not wish to participate in the research study could complete a standard incident report form as was consistent with hospital policy. Thus, medical errors were only reported once, either through the anonymous system for study participants or on incident reports for those who did not wish to participate in the project.

Before and during the data collection period, a member of the research team met with physicians on duty in the study units, including residents, fellows, and attending physicians, to explain the study procedures. Clinical nurse specialists in the study units provided the nursing staff with ongoing training based on a curriculum prepared specifically for the project. Topics covered in the training of both nurses and physicians included accessing the system, examples of medical errors, the importance of reporting errors, including near‐misses, and types of feedback provided. The anonymous nature of the reports was stressed, and the review procedures were explained.

During the study, nurses and physicians accessed the report form by clicking on an icon on a workstation desktop. The reporter was asked to provide the date and time when and the unit on which the event occurred. After filling in this information, the 2 dialog boxes on the form had to be completed. On the first, the reporter was asked to describe the event and on the second to report the outcome, if known, of the patient involved. All information on the form was completed using free text; there were no pull‐down menus or radio buttons. This was done to encourage more complete narratives and to be as inclusive as possible when asking nurses and physicians to report. Prior to the study, it was believed that asking potential reporters to classify whether events were errors or to classify them by type or other characterizations might keep nurses and physicians from reporting events that did not fit into a particular category and that a forced entry format would tend to reinforce current biases about errors rather than maximize the amount of new information gathered. Finally, to preserve anonymity, reporters were not asked to give any information about themselves, including profession (nurse or physician). However, they could provide their own names if they wanted feedback on the event, with the obvious loss of anonymity. Once the form was completed, the physician or nurse clicked the submit button to transmit the report to the research team.

A member of the research team reviewed every anonymous report within 48 hours of submission. If the event described was considered a medical error with the potential for serious patient injury, the investigator contacted a member of the clinical leadership of the unit (consisting of a medical director, one or more head nurses, and clinical nurse specialists) about the report. Every month members of the clinical leadership also received batched copies of all reports from their unit. Otherwise, neither the clinical nor the administrative leadership had access to the reports.

Each of the study's 3 pediatrician investigators (J.T., D.B., and E.K.) independently reviewed every report. First, the reviewer determined whether the event described constituted a medical error based on the definition provided by the Institute of Medicine.1 Events were further categorized by severity, occurrence to patient, and type. A medical error was considered serious if it resulted in or had the potential to result in permanent patient injury or death, moderately serious if it resulted in or had the potential to result in temporary physical or emotional injury, or trivial if it was unlikely to result in injury or change in treatment plan. Each error was further classified by whether it actually occurredeither as having actually happened to a patient or as being a near‐miss, an error detected before reaching the patient.

Because there is, to our knowledge, no standardized taxonomy for categorizing types of medical errors that occur in inpatient pediatric patients, a classification system was developed by the University of Washington Developmental Center for Evaluation and Research in Pediatric Patient Safety. (The developmental center and its organizational structure have been previously described).10 A preliminary classification system was patterned after the schema proposed by Leape et al. and adapted for use in pediatrics.19 After reviewing a series of incident reports for another project, the developers of this classification system for types of errors further refined it. The final taxonomy had 8 main types of medical errors, most with subtypes. The schema used for classifying types of errors in this study is shown in Table 1. Although the reviewers found frequent overlap, they determined the primary type of error for events described in each report based on this classification system. Final categorization of the errors, including severity, occurrence to patient, and type, was based on agreement by at least 2 of the 3 reviewers. In instances in which there was not sufficient agreement for categorization, the 3 reviewers reached a consensus after discussion.

For comparison, an identical review was conducted of incident reports completed in the 2 study units during the same months (mid‐February through mid‐May) in the years 1999‐2002. By including data from several previous years for comparison, the potential problem of selecting a period that was an outlier (in which one or more unusual factors led to increased or decreased reporting) was avoided. We selected the years 1999‐2002 because this was a period of increasing interest in better understanding medical errors at CHRMC. During this period, physicians and staff were encouraged to report medical errors, including near‐miss events, on incident reports. As with the anonymous electronic submissions, each investigator independently reviewed all the selected incident reports, with final classification based on the same schema used for the anonymous reports.

Comparison of the 2 reporting systems was complicated by the hospitalwide quality improvement program to increase the accuracy of labeling laboratory specimens that was ongoing during 1999‐2002. As part of this program, the hospital staff was encouraged to use the incident report system to document unlabeled or mismatched laboratory specimens and patients without proper identification from whom a laboratory specimen was to be obtained (eg, missing a hospital identification bracelet). Laboratory personnel completed most of these incident reports. In a previous review of incident report data from CHRMC, we found that 35% of medical errors reported were related to improper labeling of laboratory specimens (unpublished data). Although reporting these events may have been helpful for monitoring progress in quality improvement, many of the events described were extremely trivial in nature. Inclusion of this one specific type of event so skewed the overall number of medical errors reported that meaningful analysis of the types, relative frequencies, and reporting of errors was difficult. Based on this experience, we considered excluding this type of event from the analysis in the current study if it constituted a significant proportion of the medical errors conveyed in incident reports. Descriptions of mislabeled lab specimens or patients without identification bracelets constituted 33.8% of all incident reports from the 2 study units; no such events were described in submissions through the anonymous reporting system.

To compare the electronic anonymous and incident‐report error reporting systems, first the number of errors reported with each system was divided by the total number of patient‐days during which data were collected in the 2 units. Rates are expressed as the number of errors per 100 patient‐days. Rate ratios (RRs) with 95% confidence intervals (95% CIs) were calculated to compare the error reporting rates of the 2 reporting systems. Poisson regression was used to assess significance; a rate ratio whose 95% CI did not include 1.0 was considered statistically significant. Initial comparisons included all reports made through both systems. For subsequent comparisons, reports pertaining to mislabeled lab specimens were excluded. Error reporting rates were compared between the 2 reporting systems overall and by unit (medical unit and IICU), type, severity, and near‐miss status. In addition, to evaluate the possibility that secular trends in reporting medical errors were responsible for any observed overall differences, error reporting rates determined with the anonymous system were compared separately with incident report error rates in 1999, 2000, 2001, and 2002. Differences in the relative frequency of reporting different types of errors with the 2 systems were assessed with chi‐square tests. Kappa statistics were computed to assess the interobserver reliability of the 3 reviewers in classifying the events in the incident and anonymous reports as medical errors.

The study was approved by the Institutional Review Board of Children's Hospital and Regional Medical Center.

RESULTS

During the 3‐month study period, 146 reports were completed using the anonymous reporting system, 131 of which were classified as medical errors (89.7%). Ninety‐five errors were reported from the medical unit, and 36 were reported from the IICU. The kappa statistic for interobserver agreement in categorizing the anonymous reports as medical errors was .526. There were a total of 5420 patient‐days in the 2 units (medical service and IICU); thus, the rate of reporting medical errors via the anonymous system was 2.41/100 patient‐days (95% CI 2.02, 2.86). As shown in Table 2, the rate of errors reported in the IICU was higher than that in the medical unit. In addition to the errors reported via the anonymous system during the study period, 25 errors were reported using incident reports. Thus, the rate of reporting errors using both systems was 2.87.

A total of 633 incident reports were completed in the 2 study units during the analogous 3‐month periods in 1999‐2002, 538 of which were categorized as medical errors (85.0%). When all reports were considered, the rate of medical errors reported via the incident report system was 2.40/100 patient‐days (95% CI 2.21, 2.61). However, 17.3% of all errors reported in 1999, 37.2% of those reported in 2000, 40.2% of those in 2001, and 39.8% of those in 2002 pertained to mislabeled laboratory specimens. After excluding these reports, the overall rate of medical error reporting during 1999‐2002, calculated using incident report data, was 1.56/100‐patient days (95% CI 1.40, 1.73). The kappa statistic for interobserver agreement in classifying incident reports as medical errors was .615. Rates of error reporting in the medical unit and IICU are shown in Table 2.

After excluding reports dealing with mislabeled laboratory specimens, the error reporting rate was significantly higher using the anonymous system than using incident reports (RR 1.54, 95% CI 1.26, 1.90). The rate of reporting errors with the anonymous system was higher than those for reporting via incident reports in 1999, 2001, and 2002; there was no significant difference in reporting rates when the data collected with the anonymous system were compared with the data on errors reported via incident reports in 2000 (RR 1.26, 95% CI 0.97, 1.67; Table 2).

Much of the increased rate of reporting via the anonymous system came from the medical unit. The medical unit had an overall RR for anonymous reporting compared with incidence report submission of 1.77 (95% CI 1.31, 2.14); the rate of reporting via the anonymous system was significantly higher than via incident reports for each of the years 1999‐2002. Conversely, the rate of reporting observed in the IICU was not significantly increased (RR 1.33, 95% CI 0.89, 1.95, P = .07).

The types of errors reported with the 2 systems are summarized in Table 3. Although the overall distribution was only marginally different between the 2 systems (P = .054), a higher proportion of the errors reported via the anonymous system were medication errors (P = .019), whereas a higher percentage of errors reported with incident reports dealt with equipment failures (P = .033). The rate of reporting medication errors with the anonymous system (1.57 reports/100 patient‐days) was significantly higher than that via incident reports (0.83 reports/100 patient days, RR 1.90, 95% CI 1.44, 2.47). When compared with the individual years for which incident report data were available, the reporting rate for medication errors was significantly higher via the anonymous system than with incident reports for each of the years 1999‐2002.

The severity of medical errors reported with the 2 systems is shown in Table 4. As can be seen, errors reported via the anonymous system and in incident reports had a similar distribution of severity, with almost 80% of medical errors classified as moderately serious. The rate of reporting serious medical errors was 0.37/100 patient‐days with the anonymous system and 0.23/100 patient‐days via incident reports (RR 1.61, 95% CI 0.91, 2.76).

With the anonymous system, 25.2% of reported medical errors were near‐misses compared with 12.6% of the errors reported with the incident report system (P = .001). The rate of reporting near‐miss medical errors was 3‐fold higher with the anonymous system relative to reporting via incident reports (RR 3.10, 95% CI 1.91, 4.98) and was significantly higher than in each of the years data on incident reports were collected and in each of the 2 units. The reporting of errors that reached the patient was also significantly more frequent with the anonymous system than via incident reports; however, this increase was less pronounced (RR 1.32, 95% CI 1.05, 1.67). Among the 33 near‐miss events reported via the anonymous system were 10 medical errors categorized as serious. Six of these were related to medications, including two 10‐fold overdoses of morphine. Overall, the rate of reporting near‐miss medication errors was significantly higher with the anonymous system than with incident reports (RR 3.10, 95% CI 1.81, 5.24).

DISCUSSION

The results of this study suggest that implementation of an anonymous system was associated with a modest increase in the reporting of medical errors during the care of hospitalized children compared with reporting via a traditional incident report system. After excluding reports of mislabeled laboratory specimens, reported as part of a specific quality improvement project, the rate of errors reported with the anonymous system was approximately 54% higher than that using incident reports. The most striking upsurge in reporting observed with the anonymous system was the 3‐fold increase in reporting of near‐miss medical errors.

Because of different types of patients, lack of denominator data, different durations of observation, and, presumably, different inherent rates of errors, it is difficult to compare different anonymous reporting systems for medical errors. In one of the few studies dealing with pediatric patients, Suresh et al, evaluated a Web‐based anonymous reporting system in 54 neonatal intensive care units (NICUs).16 Over a 27‐month period, 1230 reports were completed via the system, for an average of slightly less than 1 report per NICU per month. This is substantially lower than the 12 errors per month reported from the IICU in our study using the anonymous system. In a study of a Web‐based anonymous system used by 18 ICUs in 11 hospitals, 854 reports were filed during a 12‐month period. The average rate of reporting ranged from 4.3 to 7.5 reports per ICU per month, with an overall mean of 6.5 reports per hospital per month.1415 However, unlike in our study, in which the anonymous system temporarily supplanted incident reports, only 2 of the 11 hospitals discontinued incident reporting.14 A national Web‐based system has been established for reporting medication errors. During a 2‐year period beginning in 1999, 154,816 medication errors were reported from 403 hospitals, for an average of 16 reports per hospital per month.18 This is less than the 28 medication errors reported per month with our anonymous system.

Anonymous systems based at a single institution have been associated with higher rates of reporting. In one study, approximately 68 events were reported per month during the first 16 weeks after full implementation of a hospitalwide anonymous system, compared with the average of 44 errors reported monthly in our project.17 In the study perhaps most comparable to ours, Osmon et al. reported on the use of an anonymously completed paper form used to report medical errors in an adult ICU.13 Patient safety advocates extensively described and promoted the reporting system prior to its use and while it was implemented. During the 6‐month study period, 8.93 medical events/100 patient‐days were reported with the system. This rate of reporting was 10‐fold higher than that reported via the standard reporting system used at that hospital.

In addition to rate of reporting medical errors, our study was designed to compare some aspects of the content of anonymous and incident reports. No statistically significant difference was found in the severity of the events reported; the rate of reporting serious medical errors was comparable between the 2 systems. This might suggest serious errors are the most likely to be reported regardless of the system used. However, given the modest number of serious events reported with either the anonymous or the incident report system (20 and 51, respectively), the power to detect a significant difference in rates was limited. Conversely, implementation of the anonymous system was associated with increased reporting of near‐miss events of all types and was a particularly useful mechanism for reporting near‐miss medication errors. Because near‐miss events may not be detected by other methods for identifying medication errors such as chart review or search for specific triggers, the use of an anonymous system may be an important tool in a multifaceted effort to improve medication safety. Perhaps the best use of an online anonymous system would be to provide a mechanism for rapid reporting of near‐miss errors, whereas other systems, such as incident reports, could be used to report errors that reach the patient.

We were surprised that although the reporting of medical errors was increased on the medical unit with the implementation of the anonymous system, there was no significant change in overall reporting in the IICU. This was possibly because reporting via incident reports was already more complete in the IICU, so that a small increase with the anonymous system was less likely to be detected However, it is equally plausible that because of the severity of illness of the patients in the IICU, physicians and staff in this unit had a perception that they did not have enough free time to report all errors. Finally, it is possible that the staff and/or clinical leadership in the medical unit was more enthusiastic about the anonymous system. Regardless, this result suggests that despite training on reporting, provision of an easy‐to‐use system, and the guarantee of anonymity, significant barriers to reporting medical errors remain.

The Kappa statistic of .526 for level of agreement between reviewers in categorizing events described with the anonymous system as medical errors indicates only a good level of agreement.20 This lack of agreement may be in part a result of the limited amount of information provided in some of the narrative reports of events. Because anonymous reports did not include names of patients or providers, it was impossible to review medical records or other information to gain additional information about the events described. However, as pointed out by others, determination of when a medical error has occurred, although seemingly simple, is frequently much less clear when reviewing actual events.21

The findings in our study should be interpreted cautiously. Because of the need for a unified system to record events across the entire hospital, anonymous reports supplanted incident reports in the 2 study units for only a 3‐month period; it is impossible to predict the long‐term trends in reporting with this system. We selected the winterspring period for the study because it is a busy time of year for children's hospitals. Rates of reporting and medical errors may change dramatically during other times of the year, particularly in a teaching hospital. An underlying assumption of our comparisons between the 2 reporting systems was that the actual rate of medical errors was unchanged throughout the period and that the differences observed were a result of more complete reporting with the anonymous system. The increased rate of reporting of medical errors found with the anonymous reporting system might have been influenced by the training given the medical personnel. It is also possible that the increased reporting rates with the anonymous system occurred because of increased publicity, both in the press and in the hospital, about medical errors and patient safety, in general. However, because there was no definite secular trend in reporting observed during the years 1999‐2002, it is unlikely that this explains our findings. Finally, it is impossible to measure the relative impact of the increased ease of reporting with the online system versus the anonymity provided.

Although the anonymous system was associated with a 54% increase in rate of reporting, it is clear that the vast majority of medical errors were not reported. If the estimates that incident reports capture 1%‐10% of errors are accurate,8, 9 the increase in reporting that we observed with the anonymous system would indicate that 1.5%15% of errors were reported. The impressive 10‐fold increase in reporting observed by Osmon et al. in their study of an anonymous system was partly a result of the very low rate of reporting with their traditional system (approximately .67 reports of medical errors/100 adult ICU patient‐days).13 A common feature of studies of anonymous systems with higher rates of reporting medical errors is the continuing presence of on‐site patient safety investigators and advocates.13, 17 Rather than the particulars of the reporting system used, this on‐site presence and advocacy may be the most important element in increasing voluntary reporting of medical errors. In our study it is likely that some of the increase in reporting observed with the anonymous system was related to publicity about the system and ongoing promotion of the importance of reporting errors by the research team.

Since completion of the study, CHRMC has been using incident reports as the main tool for collecting data on medical errors in all units. However, based on our experiences, a new reporting tool, called e‐feedback, has been instituted. The goal of this system is to allow physicians and staff members to quickly report events that may be indicative of systems problems in the delivery of care. The reports are reviewed by designated multidisciplinary teams in various units throughout the hospital so that changes can be implemented, if needed.

CONCLUSIONS

Although there was a modest increase in the number of reports, the results of this study indicate that the implementation of an anonymous online reporting system (with training on the use of the system) was not a panacea for the problem of underreporting of medical error. Use of a system such as we have described may be an effective tool for increasing the reporting of near‐miss events., However, our results suggest that methodologies in addition to voluntary or semivoluntary reporting systems are needed to more fully collect information on medical errors.