Evan M. Benjamin, MD

Patient experience has become a major component of the Center for Medicare and Medicaid Services Value‐Based Purchasing initiative.[1] Hospitals have therefore focused quality improvement (QI) efforts on this area.[2] Hospital performance in the realm of patient experience is generally determined using systematic surveys with closed‐ended questions, but patient‐generated narrative feedback can help hospitals identify the components of care that contribute to patient satisfaction and or are in need of improvement.[3] Online narrative responses posted by patients on rating websites or social media have been criticized because they may not be representative of the population,[4] but they also have some advantages.[5] Any patient may leave a comment, not just those who are selected for a survey. Patients may also experience benefits through the act of sharing their story with others. Moreover, most US hospitals use some form of social media,[6] which they can theoretically use to self‐collect narrative data online. To realize the full potential of patient‐generated online narratives, we need a clearer understanding of the best practices for collecting and using these narratives. We therefore solicited patient feedback on the Facebook page of a large tertiary academic medical center to determine whether it is feasible to use social media platforms for learning about and improving hospital quality.

METHODS

Baystate Medical Center (BMC) is a tertiary care medical center in western Massachusetts. We identified key BMC stakeholders in the areas of QI and public affairs. Noting that patients have expressed interest in leaving comments via social media,[7] the group opted to perform a pilot study to obtain patient narratives via a Facebook prompt (Facebook is a social media site used by an estimated 58% of US adults[8]). The BMC public affairs department delivered a press release to the local media describing a 3‐week period during which patients were invited to leave narrative feedback on the BMC Facebook wall. The BMC Institutional Review Board deemed that this study did not constitute human subjects research.

During March 2014 (March 10, 2014March 24, 2014), we posted (once a week) an open‐ended prompt on BMC's Facebook wall. The prompt was designed to elicit novel descriptions of patient experience that could help to drive QI. It read: We want to hear about your experiences. In the comment section below, please tell us what we do well and how we can improve your care. Because of concerns about the potential reputational risks of allowing open feedback on a public social media page, the prompt also reminded patients of the social media ground rules: there should be no mention of specific physicians, nurses, or other caregivers by name (for liability reasons); and patients should not include details about their medical history (for privacy reasons).

We collected all posts to preserve comments and used directed qualitative content analysis to examine them.[9] Two research team members[3, 10, 11] independently coded the responses. Starting with an a priori codebook that was developed during a previous study,[3] they amended the codebook through an iterative process to incorporate new concepts. After independently coding all blocks of text, the coders reviewed their coding selections and resolved discrepancies through discussion. We then performed second‐level coding, in which codes were organized into major pertinent themes. We reviewed the coded text after applying secondary codes in order to check for accuracy of coding and theme assignment as well as completeness of second‐level coding. We calculated percent agreement, defined as both raters scoring a block of text with the same code divided by total number of codes. We also calculated the Spearman correlation between the 2 reviewers. We used descriptive statistics to assess the frequency of select codes and themes (see Supporting Information, Appendix 1 and Appendix 2, in the online version of this article).[9, 12, 13]

RESULTS

Over a 3‐week study period, 47 comments were submitted by 37 respondents. This yielded 148 codable statements (Table 1). Despite limited information on respondents, we ascertained from Facebook that 32 (86%) were women and 5 (14%) were men.

From coded text, several broad themes were identified (see Table 1 for representative quotes): (1) comments about staff (17/37 respondents, 45.9%). These included positive descriptions of efficiency, caring behavior, good training, and good communication, whereas negative comments included descriptions of unfriendliness, apparent lack of caring, inattentiveness, poor training, unprofessional behavior, and poor communication; (2) comments about specific departments (22/37, 59.5%); (3) comments on technical aspects of care, including perceived errors, incorrect diagnoses, and inattention to pain control (9/37, 24.3%); and (4) comments describing the hospital physical plant, parking, and amenities (9/37, 24.3%). There were a few miscellaneous comments that did not fit into these broad themes, such as expressions of gratitude for our solicitation of narratives. Percent agreement between coders was 80% and Spearman's Rho was 0.82 (p<0.001).

A small number (n=3) of respondents repeatedly made comments over the 3‐week period, accounting for 30% (45/148) of codes. These repetitive commenters tended to dominate the Facebook conversation, at times describing the same experience more than once.

DISCUSSION

In this study evaluating the potential utility of social media as a hospital QI tool, several broad themes emerged. From these themes, we identified several areas that could be deemed as QI targets, including: training staff to be more responsive and sensitive to patients needs and concerns, improving patient and visitor parking, and reducing emergency department waiting times. However, the insight gained from solicited Facebook comments was similar to feedback gained from more traditional approaches of soliciting patient perspectives on care, such as patient experience surveys.[14]

Our findings should be viewed in the context of prior work focused on patient narratives in healthcare. Greaves et al. used sentiment analysis to describe the content of nearly 200,000 tweets (comments posted on the social networking website Twitter) sent to National Health Service (NHS) hospitals.[15] Themes were similar to those found in our study: (1) interaction with staff, (2) environment and facilities, and (3) issues of access and timeliness of service. Notably, these themes mirrored prior work examining narratives at NHS hospitals[3] and were similar to domains of commonly used surveys of patient experience.[14] The authors noted that there were issues with the signal to noise ratio (only about 10% of tweets were about quality) and the enforced brevity of Twitter (tweets must be 140 characters or less). These limitations suggest that using Twitter to identify QI targets would be difficult.

In contrast to Greaves et al., we chose to solicit feedback on our hospital's Facebook page. Facebook does not have Twitter's enforced brevity, allowing for more detailed narratives. In addition, we did not encounter the signal‐to‐noise problem, because our prompt was designed to request feedback that was relevant to recent experiences of care. However, a few respondents dominated the conversation, supporting the hypothesis that those most likely to comment may be the patients or families who have had the best or worst experiences. In the future, we will attempt to address this limitation and reduce the influence of repeat commenters by changing our prompt (eg, Please tell us about your experience, but please do not post descriptions of the same experience more than once.).

This pilot demonstrated some of the previously described benefits of online narratives.[5] First, there appears to be value in allowing patients to share their experiences and to read the experiences of others (as indicated in a few grateful patients comments). Second, soliciting online narratives offers a way for hospitals to demonstrate a commitment to transparency. Third, in contrast to closed‐ended survey questions, narrative comments help to identify why patients were satisfied or unsatisfied with their care. Although some surveys with closed‐ended questions also allow for narratives, these comments may or may not be carefully reviewed by the hospital. Using social media to solicit and respond to comments enhances existing methods for evaluating patient experience by engaging patients in a public space, which increases the likelihood that hospitals will attempt to improve care in response.

Notably, none of the identified areas for improvement could be considered novel QI targets for BMC. For example, our hospital has been very focused on training staff around patient experience, and emergency department wait times are the focus of a system‐wide improvement effort called Patient Progress.

This study has other limitations. We conducted this study over a 3‐week time period in a single center and on a single social media site whose members may not be representative of the overall patient population at BMC. Although we do not know how generalizable our findings are (in terms of identifying QI targets), we feel that we have demonstrated how using social media to collect data on patient experience is feasible and could be informative for other hospitals in other locations. It is possible that we did not allow the experiment to run long enough; a longer time or broader outreach (eg, a handout given to every discharged patient over a longer period) may be needed to allow patients adequate opportunity to comment. Of note, we did not specifically examine responses by time period, but it does seem, in post hoc analysis, that after 2 weeks of data collection we reached theoretical saturation with no new themes emerging in the third week (eg, third‐week comments included I heart your nurses. and Love Baystate but hate the ER.). More work is also needed that includes a broader range of social media platforms and more participating hospitals.

In conclusion, the opportunity to provide feedback on Facebook has the potential to engage and empower patients, and hospitals can use these online narratives to help to drive improvement efforts. Yet potential benefits must be weighed against reputational risks, a lack of representative respondents, and the paucity of novel QI targets obtained in this study.

Disclosures: Dr. Lagu is supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under award number K01HL114745. The authors report no conflicts of interest.

Patient experience has become a major component of the Center for Medicare and Medicaid Services Value‐Based Purchasing initiative.[1] Hospitals have therefore focused quality improvement (QI) efforts on this area.[2] Hospital performance in the realm of patient experience is generally determined using systematic surveys with closed‐ended questions, but patient‐generated narrative feedback can help hospitals identify the components of care that contribute to patient satisfaction and or are in need of improvement.[3] Online narrative responses posted by patients on rating websites or social media have been criticized because they may not be representative of the population,[4] but they also have some advantages.[5] Any patient may leave a comment, not just those who are selected for a survey. Patients may also experience benefits through the act of sharing their story with others. Moreover, most US hospitals use some form of social media,[6] which they can theoretically use to self‐collect narrative data online. To realize the full potential of patient‐generated online narratives, we need a clearer understanding of the best practices for collecting and using these narratives. We therefore solicited patient feedback on the Facebook page of a large tertiary academic medical center to determine whether it is feasible to use social media platforms for learning about and improving hospital quality.

METHODS

Baystate Medical Center (BMC) is a tertiary care medical center in western Massachusetts. We identified key BMC stakeholders in the areas of QI and public affairs. Noting that patients have expressed interest in leaving comments via social media,[7] the group opted to perform a pilot study to obtain patient narratives via a Facebook prompt (Facebook is a social media site used by an estimated 58% of US adults[8]). The BMC public affairs department delivered a press release to the local media describing a 3‐week period during which patients were invited to leave narrative feedback on the BMC Facebook wall. The BMC Institutional Review Board deemed that this study did not constitute human subjects research.

During March 2014 (March 10, 2014March 24, 2014), we posted (once a week) an open‐ended prompt on BMC's Facebook wall. The prompt was designed to elicit novel descriptions of patient experience that could help to drive QI. It read: We want to hear about your experiences. In the comment section below, please tell us what we do well and how we can improve your care. Because of concerns about the potential reputational risks of allowing open feedback on a public social media page, the prompt also reminded patients of the social media ground rules: there should be no mention of specific physicians, nurses, or other caregivers by name (for liability reasons); and patients should not include details about their medical history (for privacy reasons).

We collected all posts to preserve comments and used directed qualitative content analysis to examine them.[9] Two research team members[3, 10, 11] independently coded the responses. Starting with an a priori codebook that was developed during a previous study,[3] they amended the codebook through an iterative process to incorporate new concepts. After independently coding all blocks of text, the coders reviewed their coding selections and resolved discrepancies through discussion. We then performed second‐level coding, in which codes were organized into major pertinent themes. We reviewed the coded text after applying secondary codes in order to check for accuracy of coding and theme assignment as well as completeness of second‐level coding. We calculated percent agreement, defined as both raters scoring a block of text with the same code divided by total number of codes. We also calculated the Spearman correlation between the 2 reviewers. We used descriptive statistics to assess the frequency of select codes and themes (see Supporting Information, Appendix 1 and Appendix 2, in the online version of this article).[9, 12, 13]

RESULTS

Over a 3‐week study period, 47 comments were submitted by 37 respondents. This yielded 148 codable statements (Table 1). Despite limited information on respondents, we ascertained from Facebook that 32 (86%) were women and 5 (14%) were men.

From coded text, several broad themes were identified (see Table 1 for representative quotes): (1) comments about staff (17/37 respondents, 45.9%). These included positive descriptions of efficiency, caring behavior, good training, and good communication, whereas negative comments included descriptions of unfriendliness, apparent lack of caring, inattentiveness, poor training, unprofessional behavior, and poor communication; (2) comments about specific departments (22/37, 59.5%); (3) comments on technical aspects of care, including perceived errors, incorrect diagnoses, and inattention to pain control (9/37, 24.3%); and (4) comments describing the hospital physical plant, parking, and amenities (9/37, 24.3%). There were a few miscellaneous comments that did not fit into these broad themes, such as expressions of gratitude for our solicitation of narratives. Percent agreement between coders was 80% and Spearman's Rho was 0.82 (p<0.001).

A small number (n=3) of respondents repeatedly made comments over the 3‐week period, accounting for 30% (45/148) of codes. These repetitive commenters tended to dominate the Facebook conversation, at times describing the same experience more than once.

DISCUSSION

In this study evaluating the potential utility of social media as a hospital QI tool, several broad themes emerged. From these themes, we identified several areas that could be deemed as QI targets, including: training staff to be more responsive and sensitive to patients needs and concerns, improving patient and visitor parking, and reducing emergency department waiting times. However, the insight gained from solicited Facebook comments was similar to feedback gained from more traditional approaches of soliciting patient perspectives on care, such as patient experience surveys.[14]

Our findings should be viewed in the context of prior work focused on patient narratives in healthcare. Greaves et al. used sentiment analysis to describe the content of nearly 200,000 tweets (comments posted on the social networking website Twitter) sent to National Health Service (NHS) hospitals.[15] Themes were similar to those found in our study: (1) interaction with staff, (2) environment and facilities, and (3) issues of access and timeliness of service. Notably, these themes mirrored prior work examining narratives at NHS hospitals[3] and were similar to domains of commonly used surveys of patient experience.[14] The authors noted that there were issues with the signal to noise ratio (only about 10% of tweets were about quality) and the enforced brevity of Twitter (tweets must be 140 characters or less). These limitations suggest that using Twitter to identify QI targets would be difficult.

In contrast to Greaves et al., we chose to solicit feedback on our hospital's Facebook page. Facebook does not have Twitter's enforced brevity, allowing for more detailed narratives. In addition, we did not encounter the signal‐to‐noise problem, because our prompt was designed to request feedback that was relevant to recent experiences of care. However, a few respondents dominated the conversation, supporting the hypothesis that those most likely to comment may be the patients or families who have had the best or worst experiences. In the future, we will attempt to address this limitation and reduce the influence of repeat commenters by changing our prompt (eg, Please tell us about your experience, but please do not post descriptions of the same experience more than once.).

This pilot demonstrated some of the previously described benefits of online narratives.[5] First, there appears to be value in allowing patients to share their experiences and to read the experiences of others (as indicated in a few grateful patients comments). Second, soliciting online narratives offers a way for hospitals to demonstrate a commitment to transparency. Third, in contrast to closed‐ended survey questions, narrative comments help to identify why patients were satisfied or unsatisfied with their care. Although some surveys with closed‐ended questions also allow for narratives, these comments may or may not be carefully reviewed by the hospital. Using social media to solicit and respond to comments enhances existing methods for evaluating patient experience by engaging patients in a public space, which increases the likelihood that hospitals will attempt to improve care in response.

Notably, none of the identified areas for improvement could be considered novel QI targets for BMC. For example, our hospital has been very focused on training staff around patient experience, and emergency department wait times are the focus of a system‐wide improvement effort called Patient Progress.

This study has other limitations. We conducted this study over a 3‐week time period in a single center and on a single social media site whose members may not be representative of the overall patient population at BMC. Although we do not know how generalizable our findings are (in terms of identifying QI targets), we feel that we have demonstrated how using social media to collect data on patient experience is feasible and could be informative for other hospitals in other locations. It is possible that we did not allow the experiment to run long enough; a longer time or broader outreach (eg, a handout given to every discharged patient over a longer period) may be needed to allow patients adequate opportunity to comment. Of note, we did not specifically examine responses by time period, but it does seem, in post hoc analysis, that after 2 weeks of data collection we reached theoretical saturation with no new themes emerging in the third week (eg, third‐week comments included I heart your nurses. and Love Baystate but hate the ER.). More work is also needed that includes a broader range of social media platforms and more participating hospitals.

In conclusion, the opportunity to provide feedback on Facebook has the potential to engage and empower patients, and hospitals can use these online narratives to help to drive improvement efforts. Yet potential benefits must be weighed against reputational risks, a lack of representative respondents, and the paucity of novel QI targets obtained in this study.

Disclosures: Dr. Lagu is supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under award number K01HL114745. The authors report no conflicts of interest.

Patient experience has become a major component of the Center for Medicare and Medicaid Services Value‐Based Purchasing initiative.[1] Hospitals have therefore focused quality improvement (QI) efforts on this area.[2] Hospital performance in the realm of patient experience is generally determined using systematic surveys with closed‐ended questions, but patient‐generated narrative feedback can help hospitals identify the components of care that contribute to patient satisfaction and or are in need of improvement.[3] Online narrative responses posted by patients on rating websites or social media have been criticized because they may not be representative of the population,[4] but they also have some advantages.[5] Any patient may leave a comment, not just those who are selected for a survey. Patients may also experience benefits through the act of sharing their story with others. Moreover, most US hospitals use some form of social media,[6] which they can theoretically use to self‐collect narrative data online. To realize the full potential of patient‐generated online narratives, we need a clearer understanding of the best practices for collecting and using these narratives. We therefore solicited patient feedback on the Facebook page of a large tertiary academic medical center to determine whether it is feasible to use social media platforms for learning about and improving hospital quality.

METHODS

Baystate Medical Center (BMC) is a tertiary care medical center in western Massachusetts. We identified key BMC stakeholders in the areas of QI and public affairs. Noting that patients have expressed interest in leaving comments via social media,[7] the group opted to perform a pilot study to obtain patient narratives via a Facebook prompt (Facebook is a social media site used by an estimated 58% of US adults[8]). The BMC public affairs department delivered a press release to the local media describing a 3‐week period during which patients were invited to leave narrative feedback on the BMC Facebook wall. The BMC Institutional Review Board deemed that this study did not constitute human subjects research.

During March 2014 (March 10, 2014March 24, 2014), we posted (once a week) an open‐ended prompt on BMC's Facebook wall. The prompt was designed to elicit novel descriptions of patient experience that could help to drive QI. It read: We want to hear about your experiences. In the comment section below, please tell us what we do well and how we can improve your care. Because of concerns about the potential reputational risks of allowing open feedback on a public social media page, the prompt also reminded patients of the social media ground rules: there should be no mention of specific physicians, nurses, or other caregivers by name (for liability reasons); and patients should not include details about their medical history (for privacy reasons).

We collected all posts to preserve comments and used directed qualitative content analysis to examine them.[9] Two research team members[3, 10, 11] independently coded the responses. Starting with an a priori codebook that was developed during a previous study,[3] they amended the codebook through an iterative process to incorporate new concepts. After independently coding all blocks of text, the coders reviewed their coding selections and resolved discrepancies through discussion. We then performed second‐level coding, in which codes were organized into major pertinent themes. We reviewed the coded text after applying secondary codes in order to check for accuracy of coding and theme assignment as well as completeness of second‐level coding. We calculated percent agreement, defined as both raters scoring a block of text with the same code divided by total number of codes. We also calculated the Spearman correlation between the 2 reviewers. We used descriptive statistics to assess the frequency of select codes and themes (see Supporting Information, Appendix 1 and Appendix 2, in the online version of this article).[9, 12, 13]

RESULTS

Over a 3‐week study period, 47 comments were submitted by 37 respondents. This yielded 148 codable statements (Table 1). Despite limited information on respondents, we ascertained from Facebook that 32 (86%) were women and 5 (14%) were men.

From coded text, several broad themes were identified (see Table 1 for representative quotes): (1) comments about staff (17/37 respondents, 45.9%). These included positive descriptions of efficiency, caring behavior, good training, and good communication, whereas negative comments included descriptions of unfriendliness, apparent lack of caring, inattentiveness, poor training, unprofessional behavior, and poor communication; (2) comments about specific departments (22/37, 59.5%); (3) comments on technical aspects of care, including perceived errors, incorrect diagnoses, and inattention to pain control (9/37, 24.3%); and (4) comments describing the hospital physical plant, parking, and amenities (9/37, 24.3%). There were a few miscellaneous comments that did not fit into these broad themes, such as expressions of gratitude for our solicitation of narratives. Percent agreement between coders was 80% and Spearman's Rho was 0.82 (p<0.001).

A small number (n=3) of respondents repeatedly made comments over the 3‐week period, accounting for 30% (45/148) of codes. These repetitive commenters tended to dominate the Facebook conversation, at times describing the same experience more than once.

DISCUSSION

In this study evaluating the potential utility of social media as a hospital QI tool, several broad themes emerged. From these themes, we identified several areas that could be deemed as QI targets, including: training staff to be more responsive and sensitive to patients needs and concerns, improving patient and visitor parking, and reducing emergency department waiting times. However, the insight gained from solicited Facebook comments was similar to feedback gained from more traditional approaches of soliciting patient perspectives on care, such as patient experience surveys.[14]

Our findings should be viewed in the context of prior work focused on patient narratives in healthcare. Greaves et al. used sentiment analysis to describe the content of nearly 200,000 tweets (comments posted on the social networking website Twitter) sent to National Health Service (NHS) hospitals.[15] Themes were similar to those found in our study: (1) interaction with staff, (2) environment and facilities, and (3) issues of access and timeliness of service. Notably, these themes mirrored prior work examining narratives at NHS hospitals[3] and were similar to domains of commonly used surveys of patient experience.[14] The authors noted that there were issues with the signal to noise ratio (only about 10% of tweets were about quality) and the enforced brevity of Twitter (tweets must be 140 characters or less). These limitations suggest that using Twitter to identify QI targets would be difficult.

In contrast to Greaves et al., we chose to solicit feedback on our hospital's Facebook page. Facebook does not have Twitter's enforced brevity, allowing for more detailed narratives. In addition, we did not encounter the signal‐to‐noise problem, because our prompt was designed to request feedback that was relevant to recent experiences of care. However, a few respondents dominated the conversation, supporting the hypothesis that those most likely to comment may be the patients or families who have had the best or worst experiences. In the future, we will attempt to address this limitation and reduce the influence of repeat commenters by changing our prompt (eg, Please tell us about your experience, but please do not post descriptions of the same experience more than once.).

This pilot demonstrated some of the previously described benefits of online narratives.[5] First, there appears to be value in allowing patients to share their experiences and to read the experiences of others (as indicated in a few grateful patients comments). Second, soliciting online narratives offers a way for hospitals to demonstrate a commitment to transparency. Third, in contrast to closed‐ended survey questions, narrative comments help to identify why patients were satisfied or unsatisfied with their care. Although some surveys with closed‐ended questions also allow for narratives, these comments may or may not be carefully reviewed by the hospital. Using social media to solicit and respond to comments enhances existing methods for evaluating patient experience by engaging patients in a public space, which increases the likelihood that hospitals will attempt to improve care in response.

Notably, none of the identified areas for improvement could be considered novel QI targets for BMC. For example, our hospital has been very focused on training staff around patient experience, and emergency department wait times are the focus of a system‐wide improvement effort called Patient Progress.

This study has other limitations. We conducted this study over a 3‐week time period in a single center and on a single social media site whose members may not be representative of the overall patient population at BMC. Although we do not know how generalizable our findings are (in terms of identifying QI targets), we feel that we have demonstrated how using social media to collect data on patient experience is feasible and could be informative for other hospitals in other locations. It is possible that we did not allow the experiment to run long enough; a longer time or broader outreach (eg, a handout given to every discharged patient over a longer period) may be needed to allow patients adequate opportunity to comment. Of note, we did not specifically examine responses by time period, but it does seem, in post hoc analysis, that after 2 weeks of data collection we reached theoretical saturation with no new themes emerging in the third week (eg, third‐week comments included I heart your nurses. and Love Baystate but hate the ER.). More work is also needed that includes a broader range of social media platforms and more participating hospitals.

In conclusion, the opportunity to provide feedback on Facebook has the potential to engage and empower patients, and hospitals can use these online narratives to help to drive improvement efforts. Yet potential benefits must be weighed against reputational risks, a lack of representative respondents, and the paucity of novel QI targets obtained in this study.

Disclosures: Dr. Lagu is supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under award number K01HL114745. The authors report no conflicts of interest.

Acknowledging striking deficiencies in the quality and safety of healthcare, the Institute of Medicine, policy makers, and payors have called for transformation of the US healthcare system.1 Public reporting of hospital performance is one key strategy for accelerating improvement2 and may improve quality in several ways. First, feedback about performance relative to peers may stimulate quality improvement activities by appealing to professionalism. Second, the desire to preserve one's reputation by not appearing on a list of poor performers may be a powerful incentive. Finally, patients and referring providers could use reports to select high‐quality hospitals, thereby shifting care from low‐quality to high‐quality hospitals and stimulating quality improvement efforts to maintain or enhance market share.

Almost 20 years after New York and Pennsylvania began reporting cardiac surgery outcomes,3 the evidence that public reporting improves healthcare quality is equivocal.4 Moreover, stakeholders have embraced public reporting to differing degrees. Public reporting does lead to greater engagement in quality improvement activities,58 and additional financial incentives provide modest incremental benefits.9 Purchasers, too, are starting to pay attention.10 In New York State, payors appear to contract more with high‐quality surgeons and avoid poorly performing outliers.11 Some payors are creating tiered systems, assigning higher patient copayments for hospitals with poor quality metrics. These new systems have not been rigorously studied and should raise concern among hospitals.12

In contrast to hospitals and payors, patients have been slow to embrace public reporting. In a survey of coronary artery bypass graft (CABG) patients in Pennsylvania, only 2% said that public reporting of mortality rates affected their decision making.13 Eight years later, only 11% of patients sought information about hospitals before deciding on elective major surgery,14 although a majority of patients in both studies expressed interest in the information. It is not clear whether recent proliferation of information on the internet will change patient behavior, but to date public reporting appears not to effect market share.5, 15, 16

Barriers to patients' use of public reporting include difficulty accessing the information, lack of trust, information that is not salient, and data that are difficult to interpret.17 In the absence of consensus on what or how to report, a growing number of organizations, including state and federal government, accrediting bodies, private foundations, and for‐profit companies report a variety of measures relating to structure, processes, and outcomes. Although these sites purport to target consumers, they sometimes offer conflicting information18 and are not easily interpreted by lay readers.19

To realize the benefits of public reporting, and minimize the unintended consequences, rating systems must report salient information in a way that is comprehensible to patients and trusted by the doctors who advise them. At the same time, they should be fair to hospitals and offer useful data for quality improvement. We offer 10 recommendations for improving the public reporting of healthcare quality information: 5 describing what to report and 5 detailing how it should be reported (Figure 1). We also examine 3 leading performance reporting programs to see how well they implement these recommendations.

Figure 1

Recommendations to Make Data Salient for Patients

Recommendations to Ensure That Data Reflects Hospital Quality

Discussion

The growing commitment to making hospital performance data public could transform the quality and safety of care in the US, introducing competition on quality and price and fostering informed consumer choice. To date, the promise of public reporting remains only partially fulfilled. Few hospitals have done more than comply with regulatory mandates and payer incentives, and consumers have failed to respond. To capture the full benefits of public reporting, we have made 10 recommendations to benefit patients and better engage hospitals. We suggest that reporting be patient‐centered, with an emphasis on making the data useful, meaningful, important, interpretable, and relevant. At the same time, hospitals, which are being judged on their performance, should have a level playing field, with measures that are timely, consistent, severity‐adjusted, evidence‐based, and which foster good clinical care. Of the 3 services we examined, Hospital Compare came closest to meeting these recommendations.

Although this blueprint for public reporting is easy to draft, it is challenging to implement. In particular, some of our suggestions, such as the one regarding risk adjustment, may not currently be feasible, because the complexity and cost of collecting clinical data, even in the era of electronic medical records, may be prohibitive. Until such data are readily available, it may be preferable to report nothing at all, rather than report data that are misleading. In the rush to make hospitals accountable, enthusiasm has often outstripped science,31 and several measures have had to be revised for unintended consequences.32

Any initiative to improve public reporting should have the buy‐in of all stakeholders, but particularly hospitals, which stand to benefit in several ways. By receiving regular feedback, they can focus on improving care, becoming better organizations. These improvements may be rewarded through direct compensation (pay‐for‐performance), decreased costs from complications, or increased market share. Hospitals will be more engaged if the data reflect actual quality, are adequately adjusted for severity, and acknowledge the role of chance. Otherwise, they will merely comply, or worse, look for opportunities to game the system. To succeed, public reporting needs to involve hospitals in establishing standards for reporting and validation, as well as auditing procedures to prevent fraud.33 The Hospital Quality Alliance (HQA, Washington, DC), a first step in this direction, at present has few measures. NSQIP (American College of Surgeons, Chicago, IL) is perhaps a better example of hospitals cooperating to set measurement standards to promote best‐practices. Public release of NSQIP data might accelerate progress. Alternatively, the National Quality Forum (NQF, Washington, DC) could expand its role from endorsing quality measures to include standardizing the way these measures are used in public reporting.

Still, if you build it, will they come? To date, public reporting has not been embraced by the public, despite its stated interest in the information. Several explanations could be offered. First, we may be presenting the wrong data. Process measures and mortality rates are important but represent abstract concepts for most patients. Surveys tell us that patients value most the experiences of other patients.14, 21 They want to know whether their pain will be controlled, whether the doctor will listen to them, whether the nurse will come when they call. The recent advent of the HCAHPS survey (AHRQ, Washington, DC) is another positive step. Stratifying the results by diagnosis and adding a few diagnosis‐specific questions would make HCAHPS even more valuable. Second, the data may not be readily available. Although most public reporting is done on the web, older patients who are deciding about hospitals may not have Internet access. Some reports are still proprietary, and cost could present another obstacle. Finally, even if freely‐available and patient‐centered, the results may not be interpretable by physicians, let alone patients.34

If public reporting is to succeed, it will require measures that better reflect patients' concerns. In order to collect the massive amounts of data required and present them in a timely fashion, better electronic record systems will be necessary. But these are no panacea; others have noted that the Department of Veterans Affairs, a leader in electronic records, still invests considerable time and money to review charts for NSQIP.35 Given the value that Americans place on transparency in other facets of their lives, it is clear that public reporting is here to stay. While much progress has been made over the past 5 years, additional research is needed to better measure quality from the patient's perspective, and to determine how this information can be used to help guide decision‐making, and to reward hospitals for offering the highest‐quality care.

Acknowledging striking deficiencies in the quality and safety of healthcare, the Institute of Medicine, policy makers, and payors have called for transformation of the US healthcare system.1 Public reporting of hospital performance is one key strategy for accelerating improvement2 and may improve quality in several ways. First, feedback about performance relative to peers may stimulate quality improvement activities by appealing to professionalism. Second, the desire to preserve one's reputation by not appearing on a list of poor performers may be a powerful incentive. Finally, patients and referring providers could use reports to select high‐quality hospitals, thereby shifting care from low‐quality to high‐quality hospitals and stimulating quality improvement efforts to maintain or enhance market share.

Almost 20 years after New York and Pennsylvania began reporting cardiac surgery outcomes,3 the evidence that public reporting improves healthcare quality is equivocal.4 Moreover, stakeholders have embraced public reporting to differing degrees. Public reporting does lead to greater engagement in quality improvement activities,58 and additional financial incentives provide modest incremental benefits.9 Purchasers, too, are starting to pay attention.10 In New York State, payors appear to contract more with high‐quality surgeons and avoid poorly performing outliers.11 Some payors are creating tiered systems, assigning higher patient copayments for hospitals with poor quality metrics. These new systems have not been rigorously studied and should raise concern among hospitals.12

In contrast to hospitals and payors, patients have been slow to embrace public reporting. In a survey of coronary artery bypass graft (CABG) patients in Pennsylvania, only 2% said that public reporting of mortality rates affected their decision making.13 Eight years later, only 11% of patients sought information about hospitals before deciding on elective major surgery,14 although a majority of patients in both studies expressed interest in the information. It is not clear whether recent proliferation of information on the internet will change patient behavior, but to date public reporting appears not to effect market share.5, 15, 16

Barriers to patients' use of public reporting include difficulty accessing the information, lack of trust, information that is not salient, and data that are difficult to interpret.17 In the absence of consensus on what or how to report, a growing number of organizations, including state and federal government, accrediting bodies, private foundations, and for‐profit companies report a variety of measures relating to structure, processes, and outcomes. Although these sites purport to target consumers, they sometimes offer conflicting information18 and are not easily interpreted by lay readers.19

To realize the benefits of public reporting, and minimize the unintended consequences, rating systems must report salient information in a way that is comprehensible to patients and trusted by the doctors who advise them. At the same time, they should be fair to hospitals and offer useful data for quality improvement. We offer 10 recommendations for improving the public reporting of healthcare quality information: 5 describing what to report and 5 detailing how it should be reported (Figure 1). We also examine 3 leading performance reporting programs to see how well they implement these recommendations.

Figure 1

Recommendations to Make Data Salient for Patients

Recommendations to Ensure That Data Reflects Hospital Quality

Discussion

The growing commitment to making hospital performance data public could transform the quality and safety of care in the US, introducing competition on quality and price and fostering informed consumer choice. To date, the promise of public reporting remains only partially fulfilled. Few hospitals have done more than comply with regulatory mandates and payer incentives, and consumers have failed to respond. To capture the full benefits of public reporting, we have made 10 recommendations to benefit patients and better engage hospitals. We suggest that reporting be patient‐centered, with an emphasis on making the data useful, meaningful, important, interpretable, and relevant. At the same time, hospitals, which are being judged on their performance, should have a level playing field, with measures that are timely, consistent, severity‐adjusted, evidence‐based, and which foster good clinical care. Of the 3 services we examined, Hospital Compare came closest to meeting these recommendations.

Although this blueprint for public reporting is easy to draft, it is challenging to implement. In particular, some of our suggestions, such as the one regarding risk adjustment, may not currently be feasible, because the complexity and cost of collecting clinical data, even in the era of electronic medical records, may be prohibitive. Until such data are readily available, it may be preferable to report nothing at all, rather than report data that are misleading. In the rush to make hospitals accountable, enthusiasm has often outstripped science,31 and several measures have had to be revised for unintended consequences.32

Any initiative to improve public reporting should have the buy‐in of all stakeholders, but particularly hospitals, which stand to benefit in several ways. By receiving regular feedback, they can focus on improving care, becoming better organizations. These improvements may be rewarded through direct compensation (pay‐for‐performance), decreased costs from complications, or increased market share. Hospitals will be more engaged if the data reflect actual quality, are adequately adjusted for severity, and acknowledge the role of chance. Otherwise, they will merely comply, or worse, look for opportunities to game the system. To succeed, public reporting needs to involve hospitals in establishing standards for reporting and validation, as well as auditing procedures to prevent fraud.33 The Hospital Quality Alliance (HQA, Washington, DC), a first step in this direction, at present has few measures. NSQIP (American College of Surgeons, Chicago, IL) is perhaps a better example of hospitals cooperating to set measurement standards to promote best‐practices. Public release of NSQIP data might accelerate progress. Alternatively, the National Quality Forum (NQF, Washington, DC) could expand its role from endorsing quality measures to include standardizing the way these measures are used in public reporting.

Still, if you build it, will they come? To date, public reporting has not been embraced by the public, despite its stated interest in the information. Several explanations could be offered. First, we may be presenting the wrong data. Process measures and mortality rates are important but represent abstract concepts for most patients. Surveys tell us that patients value most the experiences of other patients.14, 21 They want to know whether their pain will be controlled, whether the doctor will listen to them, whether the nurse will come when they call. The recent advent of the HCAHPS survey (AHRQ, Washington, DC) is another positive step. Stratifying the results by diagnosis and adding a few diagnosis‐specific questions would make HCAHPS even more valuable. Second, the data may not be readily available. Although most public reporting is done on the web, older patients who are deciding about hospitals may not have Internet access. Some reports are still proprietary, and cost could present another obstacle. Finally, even if freely‐available and patient‐centered, the results may not be interpretable by physicians, let alone patients.34

If public reporting is to succeed, it will require measures that better reflect patients' concerns. In order to collect the massive amounts of data required and present them in a timely fashion, better electronic record systems will be necessary. But these are no panacea; others have noted that the Department of Veterans Affairs, a leader in electronic records, still invests considerable time and money to review charts for NSQIP.35 Given the value that Americans place on transparency in other facets of their lives, it is clear that public reporting is here to stay. While much progress has been made over the past 5 years, additional research is needed to better measure quality from the patient's perspective, and to determine how this information can be used to help guide decision‐making, and to reward hospitals for offering the highest‐quality care.

Acknowledging striking deficiencies in the quality and safety of healthcare, the Institute of Medicine, policy makers, and payors have called for transformation of the US healthcare system.1 Public reporting of hospital performance is one key strategy for accelerating improvement2 and may improve quality in several ways. First, feedback about performance relative to peers may stimulate quality improvement activities by appealing to professionalism. Second, the desire to preserve one's reputation by not appearing on a list of poor performers may be a powerful incentive. Finally, patients and referring providers could use reports to select high‐quality hospitals, thereby shifting care from low‐quality to high‐quality hospitals and stimulating quality improvement efforts to maintain or enhance market share.

Almost 20 years after New York and Pennsylvania began reporting cardiac surgery outcomes,3 the evidence that public reporting improves healthcare quality is equivocal.4 Moreover, stakeholders have embraced public reporting to differing degrees. Public reporting does lead to greater engagement in quality improvement activities,58 and additional financial incentives provide modest incremental benefits.9 Purchasers, too, are starting to pay attention.10 In New York State, payors appear to contract more with high‐quality surgeons and avoid poorly performing outliers.11 Some payors are creating tiered systems, assigning higher patient copayments for hospitals with poor quality metrics. These new systems have not been rigorously studied and should raise concern among hospitals.12

In contrast to hospitals and payors, patients have been slow to embrace public reporting. In a survey of coronary artery bypass graft (CABG) patients in Pennsylvania, only 2% said that public reporting of mortality rates affected their decision making.13 Eight years later, only 11% of patients sought information about hospitals before deciding on elective major surgery,14 although a majority of patients in both studies expressed interest in the information. It is not clear whether recent proliferation of information on the internet will change patient behavior, but to date public reporting appears not to effect market share.5, 15, 16

Barriers to patients' use of public reporting include difficulty accessing the information, lack of trust, information that is not salient, and data that are difficult to interpret.17 In the absence of consensus on what or how to report, a growing number of organizations, including state and federal government, accrediting bodies, private foundations, and for‐profit companies report a variety of measures relating to structure, processes, and outcomes. Although these sites purport to target consumers, they sometimes offer conflicting information18 and are not easily interpreted by lay readers.19

To realize the benefits of public reporting, and minimize the unintended consequences, rating systems must report salient information in a way that is comprehensible to patients and trusted by the doctors who advise them. At the same time, they should be fair to hospitals and offer useful data for quality improvement. We offer 10 recommendations for improving the public reporting of healthcare quality information: 5 describing what to report and 5 detailing how it should be reported (Figure 1). We also examine 3 leading performance reporting programs to see how well they implement these recommendations.

Figure 1

Recommendations to Make Data Salient for Patients

Recommendations to Ensure That Data Reflects Hospital Quality

Discussion

The growing commitment to making hospital performance data public could transform the quality and safety of care in the US, introducing competition on quality and price and fostering informed consumer choice. To date, the promise of public reporting remains only partially fulfilled. Few hospitals have done more than comply with regulatory mandates and payer incentives, and consumers have failed to respond. To capture the full benefits of public reporting, we have made 10 recommendations to benefit patients and better engage hospitals. We suggest that reporting be patient‐centered, with an emphasis on making the data useful, meaningful, important, interpretable, and relevant. At the same time, hospitals, which are being judged on their performance, should have a level playing field, with measures that are timely, consistent, severity‐adjusted, evidence‐based, and which foster good clinical care. Of the 3 services we examined, Hospital Compare came closest to meeting these recommendations.

Although this blueprint for public reporting is easy to draft, it is challenging to implement. In particular, some of our suggestions, such as the one regarding risk adjustment, may not currently be feasible, because the complexity and cost of collecting clinical data, even in the era of electronic medical records, may be prohibitive. Until such data are readily available, it may be preferable to report nothing at all, rather than report data that are misleading. In the rush to make hospitals accountable, enthusiasm has often outstripped science,31 and several measures have had to be revised for unintended consequences.32

Any initiative to improve public reporting should have the buy‐in of all stakeholders, but particularly hospitals, which stand to benefit in several ways. By receiving regular feedback, they can focus on improving care, becoming better organizations. These improvements may be rewarded through direct compensation (pay‐for‐performance), decreased costs from complications, or increased market share. Hospitals will be more engaged if the data reflect actual quality, are adequately adjusted for severity, and acknowledge the role of chance. Otherwise, they will merely comply, or worse, look for opportunities to game the system. To succeed, public reporting needs to involve hospitals in establishing standards for reporting and validation, as well as auditing procedures to prevent fraud.33 The Hospital Quality Alliance (HQA, Washington, DC), a first step in this direction, at present has few measures. NSQIP (American College of Surgeons, Chicago, IL) is perhaps a better example of hospitals cooperating to set measurement standards to promote best‐practices. Public release of NSQIP data might accelerate progress. Alternatively, the National Quality Forum (NQF, Washington, DC) could expand its role from endorsing quality measures to include standardizing the way these measures are used in public reporting.

Still, if you build it, will they come? To date, public reporting has not been embraced by the public, despite its stated interest in the information. Several explanations could be offered. First, we may be presenting the wrong data. Process measures and mortality rates are important but represent abstract concepts for most patients. Surveys tell us that patients value most the experiences of other patients.14, 21 They want to know whether their pain will be controlled, whether the doctor will listen to them, whether the nurse will come when they call. The recent advent of the HCAHPS survey (AHRQ, Washington, DC) is another positive step. Stratifying the results by diagnosis and adding a few diagnosis‐specific questions would make HCAHPS even more valuable. Second, the data may not be readily available. Although most public reporting is done on the web, older patients who are deciding about hospitals may not have Internet access. Some reports are still proprietary, and cost could present another obstacle. Finally, even if freely‐available and patient‐centered, the results may not be interpretable by physicians, let alone patients.34

If public reporting is to succeed, it will require measures that better reflect patients' concerns. In order to collect the massive amounts of data required and present them in a timely fashion, better electronic record systems will be necessary. But these are no panacea; others have noted that the Department of Veterans Affairs, a leader in electronic records, still invests considerable time and money to review charts for NSQIP.35 Given the value that Americans place on transparency in other facets of their lives, it is clear that public reporting is here to stay. While much progress has been made over the past 5 years, additional research is needed to better measure quality from the patient's perspective, and to determine how this information can be used to help guide decision‐making, and to reward hospitals for offering the highest‐quality care.

It is widely acknowledged that the U.S. health care system is plagued by error and inefficiency and that these factors contribute to as many as 44,000‐98,000 deaths each year in U.S. hospitals. In To Err Is Human: Building a Safer Health System, the Institute of Medicine1 outlined the critical role that information technology can play in improving patient safety and highlighted computerized physician order entry (CPOE) systems for their potential to reduce the frequency of medication errors and to improve the quality of medical care.

Computerized physician order entry systems are specialized software applications that allow physicians to place orders directly into a computer. This process has a number of potential advantages over traditional handwritten ordering, including the ability to structure the ordering process to ensure the completeness of individual orders, to provide clinical decision support through diagnosis‐based order sets, and to automatically check orders for potential drugallergy, drugdrug, and drugfood interactions.2 Finally, entering orders directly into a computer eliminates the problem of transcription‐related errors that stem from the difficulty of interpreting handwriting. In clinical trials, the introduction of CPOE has been shown to reduce the frequency of medication errors, to improve the use of preventive services, and to reduce costs.36 Recognition of the benefits of these systems has not been confined to the medical community. The Leapfrog Organization, a coalition of large businesses in the United States, has chosen CPOE as one of its 3 initial safety leaps and has established a threshold that 70% of medication orders should be entered directly by physicians.7

Although the benefits of CPOE systems are widely recognized, few hospitals have implemented these systems successfully.8, 9 Those that have, have often developed the applications internally, and many have relied on house staff to do most or all of the actual ordering.10 However, most hospitals do not have the expertise for internal development and instead rely on commercially available products. Moreover, most patients hospitalized in the United States are cared for by attending physicians working without the assistance of house staff.11 In light of the importance of successfully implementing CPOE systems in such settings, we assessed the adoption of CPOE by attending physicians at 2 community hospitals where its use was voluntary and examined the characteristics and attitudes associated with use of the system to place orders.

METHODS

RESULTS

During the study period the target group of physicians placed a total of 135,692 orders, of which 69,654 (51%) were placed directly into the CPOE system, 38,878 (29%) were made using pen and paper, 7,208 (5%) were made verbally, and 19,952 (15%) were placed by telephone. Three hundred and fifty‐six (71%) of the 502 surveys sent out to physicians at the 2 hospitals were returned. Thirteen surveys were excluded from analysis because the respondent was not a physician, and 2 because we were unable to match the survey to system usage data, leaving a total of 341 surveys for analysis. Order entry rates were not computed for an additional 3 physicians who only placed verbal and telephone orders during the study period. Response rates did not differ by clinician specialty (P = .53); compared to those of nonresponders, respondents had a similar median total number of orders (111 vs. 101, P = .67) and a higher median order entry rate (66% vs. 48%, P = .03).

Characteristics and Attitudes of High, Intermediate, and Low Users

The median order entry rate of respondents was 66%. One hundred and forty‐one (42%) placed at least 80% of their orders directly into the system, whereas 109 (32%) placed no more than 20% of their orders directly in the system (Fig. 1). There was not a significant difference between the low, intermediate, and high use groups in the total number of orders that each physician placed during the study period (Table 3). Sex, years since graduation from medical school, years in practice at the study institution, and use of computers in the outpatient setting were not meaningfully different between the 3 categories of users (Table 3). On the other hand, medical specialty was strongly associated with use of the system, with anesthesiologists, pediatricians, and surgeons the specialties with the largest proportion of high users. Furthermore, physicians who were trained in a CPOE environment and those who reported daily use of computers for personal activities showed the highest levels of adoption. Physicians at Franklin Medical Center showed lower levels of order entry than their counterparts at Baystate.

Figure 1

Table 3.Characteristics of Survey Respondents (n=338) with Written and/or Direct Entry Orders in Month Preceding Survey according to Low, Intermediate, and High Usage of a CPOE System

	Low (20%) n (row %)	Intermediate (20%‐79%) n (row %)	High (80%) n (row %)	P value
a Among n = 299 with outpatient practice. b Because of missing survey responses, category values may not add up to total. c Pearson chi‐square P value. d Mantel‐Haenszel chi‐square P value. e Kruskal‐Wallis P value
	n = 109	n = 88	n = 141
Hospital				< .01c
Baystate	73 (25)	79 (27)	138 (48)
Franklin	36 (75)	9 (19)	3 (6)
Sex				.69c
Female	28 (29)	24 (25)	43 (45)
Male	81 (33)	64 (26)	98 (40)
Specialty				.0001c
Anesthesia	8 (35)	3 (13)	12 (52)
Internal medicine	45 (33)	37 (27)	53 (39)
Medicine/pediatrics	6 (46)	5 (38)	2 (15)
OB/GYN	20 (56)	12 (33)	4 (11)
Pediatrics	13 (24)	9 (17)	32 (59)
Surgery	14 (23)	21 (34)	26 (43)
Other	3 (19)	1 (6)	12 (75)
Do you use a computer in your outpatient practice?a
Yes	75 (31)	61 (25)	105 (44)	.22c
No	20 (36)	18 (33)	17 (31)
Level of personal computer useb				.045d
Rarely	11 (44)	8 (32)	6 (24)
A few times a month	7 (33)	4 (19)	10 (48)
Several times a week	28 (35)	25 (31)	28 (35)
At least once a day	62 (30)	50 (24)	97 (46)
Training at an institution that had CPOE				.037c
Yes	30 (26)	40 (34)	46 (40)
No	76 (35)	48 (22)	94 (43)
	Median (IQR)	Median (IQR)	Median (IQR)
Years since graduation from medical school	21 (16, 28)	18 (14, 25)	19 (12, 25)	.06e
Years in practice at study institution	12 (5, 19)	12 (6, 19)	12 (6, 17)	.84e
Total number of orders placed	112 (45, 306)	105 (56, 254)	113 (44, 382)	.92e

Use of the system was highly associated with physician attitudes toward CPOE, with the views of intermediate and high users consistently different than those of low users (Fig. 2). The associations found held true regardless of hospital: low, intermediate, and high users from Franklin had similar responses to those from Baystate (P > .05 for all questions), and the data from the 2 hospitals therefore were combined for presentation. Although few physicians believed that the user interface of the system supported their work flow, high and intermediate users were 3 times as likely to share this view than were low users (Q7; Fig. 2). Similarly, 19% of low users, 31% of intermediate users, and 45% of high users believed that entering orders into the system was faster than writing orders (Q1). High and intermediate users of the system were more likely than low users to believe that orders entered into the system were carried out more rapidly (Q2) and led to fewer medication (Q3) and nonmedication (Q4) errors. Regardless of their utilization pattern, most physicians believed that order sets played an important role in promoting efficiency and quality.

Figure 2

DISCUSSION

In this study of the clinical computing practices of physicians at 2 community hospitals, we observed wide variation in the adoption of CPOE by individual attendings. Although roughly one‐third rarely placed orders directly into the system, 42% had an order entry rate of at least 80%. Contrary to our initial expectation, we found little association between a physician's order entry rate with years in practice, duration of exposure to CPOE, or use of computers in the outpatient setting. On the other hand, we observed marked differences in use of the CPOE system across specialty lines and found that physicians who were exposed to CPOE during training and those who were regular users of computers for personal activities were more likely to embrace this technology. Further, we observed important differences between physicians who used the system to place some or most of their orders and those who did so only rarely in their beliefs and attitudes about the impact and benefits of CPOE. Physicians with higher order entry rates were more likely than their colleagues to believe that placing orders electronically was faster than handwriting and that use of the system led to fewer medical errors. These findings should be encouraging to hospitals hoping to implement CPOE because they suggest that successful adoption of CPOE is not limited to physicians who have just completed their residencies or to hospitals with the capability of designing and building their own systems. On the contrary, we documented that women, older physicians, and those with limited CPOE experience were as likely to be frequent users, especially if they perceived CPOE to be safer than handwriting and if they believed the user interface supported the efficient entering of orders.

On the basis of these results we recommend that in addition to purchasing systems that meet physician work‐flow needs and support the efficient entry of orders, hospital leaders should emphasize the quality and safety benefits of CPOE as part of a comprehensive change management strategy. The differences we observed in order entry rates across specialties may have resulted from several factors, including inherent differences in personality type associated with choice of specialty and in the level of customization of a system reflected in which and how many order sets are included. Such findings suggest that when it comes to CPOE, one size does not fit all, and implementation planning should be carried out at the specialty level. Finally, our observation that physicians who had exposure to CPOE during training were more likely to use the system to place orders suggests that the nation's training institutions will play an important role in fostering universal adoption of this technology.

Several earlier studies have reported on physician experiences with CPOE systems. Murff and Kannry12 surveyed 94 internal medicine house staff to compare experiences with 2 CPOE systems: the Department of Veterans Affairs Computerized Patient Record System (CPRS) and a commercially available product. They found striking differences in user satisfaction with numerous aspects of the systems, however they did not address attitudes toward safety or quality, and because house staff were required to place orders electronically they were unable to correlate responses with actual usage patterns. Weiner et al.13 compared the opinions of internal medicine house staff, attendings, fellows, and nurses about the benefits and challenges of using a computerized provider order entry system. In contrast to the findings from our study, Weiner et al. reported that more than half of physicians believed that provider order entry led to a greater number of errors, and only a minority believed the system increased quality of care overall. Finally, Lee et al.14 surveyed medical and surgical house officers and nurses at a large academic medical center about their satisfaction with a locally developed order entry system. They found that attitudes about the impact of the system on productivity and ease of use were more strongly associated with overall satisfaction than having undergone training or experience with personal computers. These findings are congruous with our own observation that beliefs about the speed with which orders are placed are closely associated with actual use of the system. They reported, as have we, that physicians placed a high value on order sets.

Our study had a number of strengths. First, we were able to offer insight into the attitudes and behaviors of a previously neglected, but critically important groupattending physicians who care for patients at community hospitals without the assistance of house staff. Second, whereas previous studies primarily assessed physician satisfaction with CPOE, we explored how physician attitudes about the impact of CPOE on work flow and on safety were associated with actual ordering habits. Information about ordering was obtained directly from the order entry system and not through self‐report. We conducted the study at 2 hospitals, a large urban community teaching hospital and a smaller rural hospital, and focused on a CPOE system that is in use at many institutions throughout the country, thereby increasing the generalizability of our findings. Although adoption of the system by physicians at the 2 hospitals differed, factors that associated with the use of CPOE to place orders were similar. Finally, we surveyed a large number of physicians, had a high response rate, and found only small differences in the utilization patterns of responders and nonresponders, suggesting that our portrayal of the attitudes of physicians was representative of the views of physicians practicing in our community.

The study had a number of weaknesses. First, we cannot be sure whether preexisting beliefs about the benefits of CPOE directly influenced physicians' use of the system or, conversely, if these attitudes developed in response to experience as users. Nevertheless, it seems practical to suggest that hospitals focus on purchasing systems that support the efficient entering of orders while simultaneously adopting a communication and change management strategy that emphasizes the safety and quality benefits of CPOE more broadly. Second, we did not attempt to validate the opinions expressed by physicians about the usability or safety benefits of the system. That said, the purpose of the study was to determine whether physician attitudes toward these issues was associated with the use of the system to place orders. Whether or not this particular CPOE system actually prevented medication errors, most physicians believed it did, a belief strongly associated with the observed order entry rates. Third, we studied a single CPOE system implemented approximately 10 years ago that does not reflect state‐of‐the‐art user interface design or functionality. Nevertheless, our observation about the importance of the user experience is probably no less relevant today. Fourth, we were unable to ascertain every order given by physicians, as some so‐called MD to RN orders may never have made it into the system. Finally, there is a small risk that some written, telephone, and verbal orders may have been randomly or systematically assigned to incorrect physicians, which would have led us to calculate inaccurate utilization rates.

CONCLUSIONS

In a voluntary community hospital environment the adoption of CPOE by attending physicians varies widely. While placing a premium on the purchase of systems that meet the work‐flow needs of physicians and support the efficient entry of orders, hospital leaders can enhance physician adoption of this technology by communicating the role of CPOE in improving quality and safety.

It is widely acknowledged that the U.S. health care system is plagued by error and inefficiency and that these factors contribute to as many as 44,000‐98,000 deaths each year in U.S. hospitals. In To Err Is Human: Building a Safer Health System, the Institute of Medicine1 outlined the critical role that information technology can play in improving patient safety and highlighted computerized physician order entry (CPOE) systems for their potential to reduce the frequency of medication errors and to improve the quality of medical care.

Computerized physician order entry systems are specialized software applications that allow physicians to place orders directly into a computer. This process has a number of potential advantages over traditional handwritten ordering, including the ability to structure the ordering process to ensure the completeness of individual orders, to provide clinical decision support through diagnosis‐based order sets, and to automatically check orders for potential drugallergy, drugdrug, and drugfood interactions.2 Finally, entering orders directly into a computer eliminates the problem of transcription‐related errors that stem from the difficulty of interpreting handwriting. In clinical trials, the introduction of CPOE has been shown to reduce the frequency of medication errors, to improve the use of preventive services, and to reduce costs.36 Recognition of the benefits of these systems has not been confined to the medical community. The Leapfrog Organization, a coalition of large businesses in the United States, has chosen CPOE as one of its 3 initial safety leaps and has established a threshold that 70% of medication orders should be entered directly by physicians.7

Although the benefits of CPOE systems are widely recognized, few hospitals have implemented these systems successfully.8, 9 Those that have, have often developed the applications internally, and many have relied on house staff to do most or all of the actual ordering.10 However, most hospitals do not have the expertise for internal development and instead rely on commercially available products. Moreover, most patients hospitalized in the United States are cared for by attending physicians working without the assistance of house staff.11 In light of the importance of successfully implementing CPOE systems in such settings, we assessed the adoption of CPOE by attending physicians at 2 community hospitals where its use was voluntary and examined the characteristics and attitudes associated with use of the system to place orders.

METHODS

RESULTS

During the study period the target group of physicians placed a total of 135,692 orders, of which 69,654 (51%) were placed directly into the CPOE system, 38,878 (29%) were made using pen and paper, 7,208 (5%) were made verbally, and 19,952 (15%) were placed by telephone. Three hundred and fifty‐six (71%) of the 502 surveys sent out to physicians at the 2 hospitals were returned. Thirteen surveys were excluded from analysis because the respondent was not a physician, and 2 because we were unable to match the survey to system usage data, leaving a total of 341 surveys for analysis. Order entry rates were not computed for an additional 3 physicians who only placed verbal and telephone orders during the study period. Response rates did not differ by clinician specialty (P = .53); compared to those of nonresponders, respondents had a similar median total number of orders (111 vs. 101, P = .67) and a higher median order entry rate (66% vs. 48%, P = .03).

Characteristics and Attitudes of High, Intermediate, and Low Users

The median order entry rate of respondents was 66%. One hundred and forty‐one (42%) placed at least 80% of their orders directly into the system, whereas 109 (32%) placed no more than 20% of their orders directly in the system (Fig. 1). There was not a significant difference between the low, intermediate, and high use groups in the total number of orders that each physician placed during the study period (Table 3). Sex, years since graduation from medical school, years in practice at the study institution, and use of computers in the outpatient setting were not meaningfully different between the 3 categories of users (Table 3). On the other hand, medical specialty was strongly associated with use of the system, with anesthesiologists, pediatricians, and surgeons the specialties with the largest proportion of high users. Furthermore, physicians who were trained in a CPOE environment and those who reported daily use of computers for personal activities showed the highest levels of adoption. Physicians at Franklin Medical Center showed lower levels of order entry than their counterparts at Baystate.

Figure 1

Table 3.Characteristics of Survey Respondents (n=338) with Written and/or Direct Entry Orders in Month Preceding Survey according to Low, Intermediate, and High Usage of a CPOE System

	Low (20%) n (row %)	Intermediate (20%‐79%) n (row %)	High (80%) n (row %)	P value
a Among n = 299 with outpatient practice. b Because of missing survey responses, category values may not add up to total. c Pearson chi‐square P value. d Mantel‐Haenszel chi‐square P value. e Kruskal‐Wallis P value
	n = 109	n = 88	n = 141
Hospital				< .01c
Baystate	73 (25)	79 (27)	138 (48)
Franklin	36 (75)	9 (19)	3 (6)
Sex				.69c
Female	28 (29)	24 (25)	43 (45)
Male	81 (33)	64 (26)	98 (40)
Specialty				.0001c
Anesthesia	8 (35)	3 (13)	12 (52)
Internal medicine	45 (33)	37 (27)	53 (39)
Medicine/pediatrics	6 (46)	5 (38)	2 (15)
OB/GYN	20 (56)	12 (33)	4 (11)
Pediatrics	13 (24)	9 (17)	32 (59)
Surgery	14 (23)	21 (34)	26 (43)
Other	3 (19)	1 (6)	12 (75)
Do you use a computer in your outpatient practice?a
Yes	75 (31)	61 (25)	105 (44)	.22c
No	20 (36)	18 (33)	17 (31)
Level of personal computer useb				.045d
Rarely	11 (44)	8 (32)	6 (24)
A few times a month	7 (33)	4 (19)	10 (48)
Several times a week	28 (35)	25 (31)	28 (35)
At least once a day	62 (30)	50 (24)	97 (46)
Training at an institution that had CPOE				.037c
Yes	30 (26)	40 (34)	46 (40)
No	76 (35)	48 (22)	94 (43)
	Median (IQR)	Median (IQR)	Median (IQR)
Years since graduation from medical school	21 (16, 28)	18 (14, 25)	19 (12, 25)	.06e
Years in practice at study institution	12 (5, 19)	12 (6, 19)	12 (6, 17)	.84e
Total number of orders placed	112 (45, 306)	105 (56, 254)	113 (44, 382)	.92e

Use of the system was highly associated with physician attitudes toward CPOE, with the views of intermediate and high users consistently different than those of low users (Fig. 2). The associations found held true regardless of hospital: low, intermediate, and high users from Franklin had similar responses to those from Baystate (P > .05 for all questions), and the data from the 2 hospitals therefore were combined for presentation. Although few physicians believed that the user interface of the system supported their work flow, high and intermediate users were 3 times as likely to share this view than were low users (Q7; Fig. 2). Similarly, 19% of low users, 31% of intermediate users, and 45% of high users believed that entering orders into the system was faster than writing orders (Q1). High and intermediate users of the system were more likely than low users to believe that orders entered into the system were carried out more rapidly (Q2) and led to fewer medication (Q3) and nonmedication (Q4) errors. Regardless of their utilization pattern, most physicians believed that order sets played an important role in promoting efficiency and quality.

Figure 2

DISCUSSION

In this study of the clinical computing practices of physicians at 2 community hospitals, we observed wide variation in the adoption of CPOE by individual attendings. Although roughly one‐third rarely placed orders directly into the system, 42% had an order entry rate of at least 80%. Contrary to our initial expectation, we found little association between a physician's order entry rate with years in practice, duration of exposure to CPOE, or use of computers in the outpatient setting. On the other hand, we observed marked differences in use of the CPOE system across specialty lines and found that physicians who were exposed to CPOE during training and those who were regular users of computers for personal activities were more likely to embrace this technology. Further, we observed important differences between physicians who used the system to place some or most of their orders and those who did so only rarely in their beliefs and attitudes about the impact and benefits of CPOE. Physicians with higher order entry rates were more likely than their colleagues to believe that placing orders electronically was faster than handwriting and that use of the system led to fewer medical errors. These findings should be encouraging to hospitals hoping to implement CPOE because they suggest that successful adoption of CPOE is not limited to physicians who have just completed their residencies or to hospitals with the capability of designing and building their own systems. On the contrary, we documented that women, older physicians, and those with limited CPOE experience were as likely to be frequent users, especially if they perceived CPOE to be safer than handwriting and if they believed the user interface supported the efficient entering of orders.

On the basis of these results we recommend that in addition to purchasing systems that meet physician work‐flow needs and support the efficient entry of orders, hospital leaders should emphasize the quality and safety benefits of CPOE as part of a comprehensive change management strategy. The differences we observed in order entry rates across specialties may have resulted from several factors, including inherent differences in personality type associated with choice of specialty and in the level of customization of a system reflected in which and how many order sets are included. Such findings suggest that when it comes to CPOE, one size does not fit all, and implementation planning should be carried out at the specialty level. Finally, our observation that physicians who had exposure to CPOE during training were more likely to use the system to place orders suggests that the nation's training institutions will play an important role in fostering universal adoption of this technology.

Several earlier studies have reported on physician experiences with CPOE systems. Murff and Kannry12 surveyed 94 internal medicine house staff to compare experiences with 2 CPOE systems: the Department of Veterans Affairs Computerized Patient Record System (CPRS) and a commercially available product. They found striking differences in user satisfaction with numerous aspects of the systems, however they did not address attitudes toward safety or quality, and because house staff were required to place orders electronically they were unable to correlate responses with actual usage patterns. Weiner et al.13 compared the opinions of internal medicine house staff, attendings, fellows, and nurses about the benefits and challenges of using a computerized provider order entry system. In contrast to the findings from our study, Weiner et al. reported that more than half of physicians believed that provider order entry led to a greater number of errors, and only a minority believed the system increased quality of care overall. Finally, Lee et al.14 surveyed medical and surgical house officers and nurses at a large academic medical center about their satisfaction with a locally developed order entry system. They found that attitudes about the impact of the system on productivity and ease of use were more strongly associated with overall satisfaction than having undergone training or experience with personal computers. These findings are congruous with our own observation that beliefs about the speed with which orders are placed are closely associated with actual use of the system. They reported, as have we, that physicians placed a high value on order sets.

Our study had a number of strengths. First, we were able to offer insight into the attitudes and behaviors of a previously neglected, but critically important groupattending physicians who care for patients at community hospitals without the assistance of house staff. Second, whereas previous studies primarily assessed physician satisfaction with CPOE, we explored how physician attitudes about the impact of CPOE on work flow and on safety were associated with actual ordering habits. Information about ordering was obtained directly from the order entry system and not through self‐report. We conducted the study at 2 hospitals, a large urban community teaching hospital and a smaller rural hospital, and focused on a CPOE system that is in use at many institutions throughout the country, thereby increasing the generalizability of our findings. Although adoption of the system by physicians at the 2 hospitals differed, factors that associated with the use of CPOE to place orders were similar. Finally, we surveyed a large number of physicians, had a high response rate, and found only small differences in the utilization patterns of responders and nonresponders, suggesting that our portrayal of the attitudes of physicians was representative of the views of physicians practicing in our community.

The study had a number of weaknesses. First, we cannot be sure whether preexisting beliefs about the benefits of CPOE directly influenced physicians' use of the system or, conversely, if these attitudes developed in response to experience as users. Nevertheless, it seems practical to suggest that hospitals focus on purchasing systems that support the efficient entering of orders while simultaneously adopting a communication and change management strategy that emphasizes the safety and quality benefits of CPOE more broadly. Second, we did not attempt to validate the opinions expressed by physicians about the usability or safety benefits of the system. That said, the purpose of the study was to determine whether physician attitudes toward these issues was associated with the use of the system to place orders. Whether or not this particular CPOE system actually prevented medication errors, most physicians believed it did, a belief strongly associated with the observed order entry rates. Third, we studied a single CPOE system implemented approximately 10 years ago that does not reflect state‐of‐the‐art user interface design or functionality. Nevertheless, our observation about the importance of the user experience is probably no less relevant today. Fourth, we were unable to ascertain every order given by physicians, as some so‐called MD to RN orders may never have made it into the system. Finally, there is a small risk that some written, telephone, and verbal orders may have been randomly or systematically assigned to incorrect physicians, which would have led us to calculate inaccurate utilization rates.

CONCLUSIONS

In a voluntary community hospital environment the adoption of CPOE by attending physicians varies widely. While placing a premium on the purchase of systems that meet the work‐flow needs of physicians and support the efficient entry of orders, hospital leaders can enhance physician adoption of this technology by communicating the role of CPOE in improving quality and safety.

It is widely acknowledged that the U.S. health care system is plagued by error and inefficiency and that these factors contribute to as many as 44,000‐98,000 deaths each year in U.S. hospitals. In To Err Is Human: Building a Safer Health System, the Institute of Medicine1 outlined the critical role that information technology can play in improving patient safety and highlighted computerized physician order entry (CPOE) systems for their potential to reduce the frequency of medication errors and to improve the quality of medical care.

Computerized physician order entry systems are specialized software applications that allow physicians to place orders directly into a computer. This process has a number of potential advantages over traditional handwritten ordering, including the ability to structure the ordering process to ensure the completeness of individual orders, to provide clinical decision support through diagnosis‐based order sets, and to automatically check orders for potential drugallergy, drugdrug, and drugfood interactions.2 Finally, entering orders directly into a computer eliminates the problem of transcription‐related errors that stem from the difficulty of interpreting handwriting. In clinical trials, the introduction of CPOE has been shown to reduce the frequency of medication errors, to improve the use of preventive services, and to reduce costs.36 Recognition of the benefits of these systems has not been confined to the medical community. The Leapfrog Organization, a coalition of large businesses in the United States, has chosen CPOE as one of its 3 initial safety leaps and has established a threshold that 70% of medication orders should be entered directly by physicians.7

Although the benefits of CPOE systems are widely recognized, few hospitals have implemented these systems successfully.8, 9 Those that have, have often developed the applications internally, and many have relied on house staff to do most or all of the actual ordering.10 However, most hospitals do not have the expertise for internal development and instead rely on commercially available products. Moreover, most patients hospitalized in the United States are cared for by attending physicians working without the assistance of house staff.11 In light of the importance of successfully implementing CPOE systems in such settings, we assessed the adoption of CPOE by attending physicians at 2 community hospitals where its use was voluntary and examined the characteristics and attitudes associated with use of the system to place orders.

METHODS

RESULTS

During the study period the target group of physicians placed a total of 135,692 orders, of which 69,654 (51%) were placed directly into the CPOE system, 38,878 (29%) were made using pen and paper, 7,208 (5%) were made verbally, and 19,952 (15%) were placed by telephone. Three hundred and fifty‐six (71%) of the 502 surveys sent out to physicians at the 2 hospitals were returned. Thirteen surveys were excluded from analysis because the respondent was not a physician, and 2 because we were unable to match the survey to system usage data, leaving a total of 341 surveys for analysis. Order entry rates were not computed for an additional 3 physicians who only placed verbal and telephone orders during the study period. Response rates did not differ by clinician specialty (P = .53); compared to those of nonresponders, respondents had a similar median total number of orders (111 vs. 101, P = .67) and a higher median order entry rate (66% vs. 48%, P = .03).

Characteristics and Attitudes of High, Intermediate, and Low Users

The median order entry rate of respondents was 66%. One hundred and forty‐one (42%) placed at least 80% of their orders directly into the system, whereas 109 (32%) placed no more than 20% of their orders directly in the system (Fig. 1). There was not a significant difference between the low, intermediate, and high use groups in the total number of orders that each physician placed during the study period (Table 3). Sex, years since graduation from medical school, years in practice at the study institution, and use of computers in the outpatient setting were not meaningfully different between the 3 categories of users (Table 3). On the other hand, medical specialty was strongly associated with use of the system, with anesthesiologists, pediatricians, and surgeons the specialties with the largest proportion of high users. Furthermore, physicians who were trained in a CPOE environment and those who reported daily use of computers for personal activities showed the highest levels of adoption. Physicians at Franklin Medical Center showed lower levels of order entry than their counterparts at Baystate.

Figure 1

Table 3.Characteristics of Survey Respondents (n=338) with Written and/or Direct Entry Orders in Month Preceding Survey according to Low, Intermediate, and High Usage of a CPOE System

	Low (20%) n (row %)	Intermediate (20%‐79%) n (row %)	High (80%) n (row %)	P value
a Among n = 299 with outpatient practice. b Because of missing survey responses, category values may not add up to total. c Pearson chi‐square P value. d Mantel‐Haenszel chi‐square P value. e Kruskal‐Wallis P value
	n = 109	n = 88	n = 141
Hospital				< .01c
Baystate	73 (25)	79 (27)	138 (48)
Franklin	36 (75)	9 (19)	3 (6)
Sex				.69c
Female	28 (29)	24 (25)	43 (45)
Male	81 (33)	64 (26)	98 (40)
Specialty				.0001c
Anesthesia	8 (35)	3 (13)	12 (52)
Internal medicine	45 (33)	37 (27)	53 (39)
Medicine/pediatrics	6 (46)	5 (38)	2 (15)
OB/GYN	20 (56)	12 (33)	4 (11)
Pediatrics	13 (24)	9 (17)	32 (59)
Surgery	14 (23)	21 (34)	26 (43)
Other	3 (19)	1 (6)	12 (75)
Do you use a computer in your outpatient practice?a
Yes	75 (31)	61 (25)	105 (44)	.22c
No	20 (36)	18 (33)	17 (31)
Level of personal computer useb				.045d
Rarely	11 (44)	8 (32)	6 (24)
A few times a month	7 (33)	4 (19)	10 (48)
Several times a week	28 (35)	25 (31)	28 (35)
At least once a day	62 (30)	50 (24)	97 (46)
Training at an institution that had CPOE				.037c
Yes	30 (26)	40 (34)	46 (40)
No	76 (35)	48 (22)	94 (43)
	Median (IQR)	Median (IQR)	Median (IQR)
Years since graduation from medical school	21 (16, 28)	18 (14, 25)	19 (12, 25)	.06e
Years in practice at study institution	12 (5, 19)	12 (6, 19)	12 (6, 17)	.84e
Total number of orders placed	112 (45, 306)	105 (56, 254)	113 (44, 382)	.92e

Use of the system was highly associated with physician attitudes toward CPOE, with the views of intermediate and high users consistently different than those of low users (Fig. 2). The associations found held true regardless of hospital: low, intermediate, and high users from Franklin had similar responses to those from Baystate (P > .05 for all questions), and the data from the 2 hospitals therefore were combined for presentation. Although few physicians believed that the user interface of the system supported their work flow, high and intermediate users were 3 times as likely to share this view than were low users (Q7; Fig. 2). Similarly, 19% of low users, 31% of intermediate users, and 45% of high users believed that entering orders into the system was faster than writing orders (Q1). High and intermediate users of the system were more likely than low users to believe that orders entered into the system were carried out more rapidly (Q2) and led to fewer medication (Q3) and nonmedication (Q4) errors. Regardless of their utilization pattern, most physicians believed that order sets played an important role in promoting efficiency and quality.

Figure 2

DISCUSSION

In this study of the clinical computing practices of physicians at 2 community hospitals, we observed wide variation in the adoption of CPOE by individual attendings. Although roughly one‐third rarely placed orders directly into the system, 42% had an order entry rate of at least 80%. Contrary to our initial expectation, we found little association between a physician's order entry rate with years in practice, duration of exposure to CPOE, or use of computers in the outpatient setting. On the other hand, we observed marked differences in use of the CPOE system across specialty lines and found that physicians who were exposed to CPOE during training and those who were regular users of computers for personal activities were more likely to embrace this technology. Further, we observed important differences between physicians who used the system to place some or most of their orders and those who did so only rarely in their beliefs and attitudes about the impact and benefits of CPOE. Physicians with higher order entry rates were more likely than their colleagues to believe that placing orders electronically was faster than handwriting and that use of the system led to fewer medical errors. These findings should be encouraging to hospitals hoping to implement CPOE because they suggest that successful adoption of CPOE is not limited to physicians who have just completed their residencies or to hospitals with the capability of designing and building their own systems. On the contrary, we documented that women, older physicians, and those with limited CPOE experience were as likely to be frequent users, especially if they perceived CPOE to be safer than handwriting and if they believed the user interface supported the efficient entering of orders.

On the basis of these results we recommend that in addition to purchasing systems that meet physician work‐flow needs and support the efficient entry of orders, hospital leaders should emphasize the quality and safety benefits of CPOE as part of a comprehensive change management strategy. The differences we observed in order entry rates across specialties may have resulted from several factors, including inherent differences in personality type associated with choice of specialty and in the level of customization of a system reflected in which and how many order sets are included. Such findings suggest that when it comes to CPOE, one size does not fit all, and implementation planning should be carried out at the specialty level. Finally, our observation that physicians who had exposure to CPOE during training were more likely to use the system to place orders suggests that the nation's training institutions will play an important role in fostering universal adoption of this technology.

Several earlier studies have reported on physician experiences with CPOE systems. Murff and Kannry12 surveyed 94 internal medicine house staff to compare experiences with 2 CPOE systems: the Department of Veterans Affairs Computerized Patient Record System (CPRS) and a commercially available product. They found striking differences in user satisfaction with numerous aspects of the systems, however they did not address attitudes toward safety or quality, and because house staff were required to place orders electronically they were unable to correlate responses with actual usage patterns. Weiner et al.13 compared the opinions of internal medicine house staff, attendings, fellows, and nurses about the benefits and challenges of using a computerized provider order entry system. In contrast to the findings from our study, Weiner et al. reported that more than half of physicians believed that provider order entry led to a greater number of errors, and only a minority believed the system increased quality of care overall. Finally, Lee et al.14 surveyed medical and surgical house officers and nurses at a large academic medical center about their satisfaction with a locally developed order entry system. They found that attitudes about the impact of the system on productivity and ease of use were more strongly associated with overall satisfaction than having undergone training or experience with personal computers. These findings are congruous with our own observation that beliefs about the speed with which orders are placed are closely associated with actual use of the system. They reported, as have we, that physicians placed a high value on order sets.

Our study had a number of strengths. First, we were able to offer insight into the attitudes and behaviors of a previously neglected, but critically important groupattending physicians who care for patients at community hospitals without the assistance of house staff. Second, whereas previous studies primarily assessed physician satisfaction with CPOE, we explored how physician attitudes about the impact of CPOE on work flow and on safety were associated with actual ordering habits. Information about ordering was obtained directly from the order entry system and not through self‐report. We conducted the study at 2 hospitals, a large urban community teaching hospital and a smaller rural hospital, and focused on a CPOE system that is in use at many institutions throughout the country, thereby increasing the generalizability of our findings. Although adoption of the system by physicians at the 2 hospitals differed, factors that associated with the use of CPOE to place orders were similar. Finally, we surveyed a large number of physicians, had a high response rate, and found only small differences in the utilization patterns of responders and nonresponders, suggesting that our portrayal of the attitudes of physicians was representative of the views of physicians practicing in our community.

The study had a number of weaknesses. First, we cannot be sure whether preexisting beliefs about the benefits of CPOE directly influenced physicians' use of the system or, conversely, if these attitudes developed in response to experience as users. Nevertheless, it seems practical to suggest that hospitals focus on purchasing systems that support the efficient entering of orders while simultaneously adopting a communication and change management strategy that emphasizes the safety and quality benefits of CPOE more broadly. Second, we did not attempt to validate the opinions expressed by physicians about the usability or safety benefits of the system. That said, the purpose of the study was to determine whether physician attitudes toward these issues was associated with the use of the system to place orders. Whether or not this particular CPOE system actually prevented medication errors, most physicians believed it did, a belief strongly associated with the observed order entry rates. Third, we studied a single CPOE system implemented approximately 10 years ago that does not reflect state‐of‐the‐art user interface design or functionality. Nevertheless, our observation about the importance of the user experience is probably no less relevant today. Fourth, we were unable to ascertain every order given by physicians, as some so‐called MD to RN orders may never have made it into the system. Finally, there is a small risk that some written, telephone, and verbal orders may have been randomly or systematically assigned to incorrect physicians, which would have led us to calculate inaccurate utilization rates.

CONCLUSIONS

In a voluntary community hospital environment the adoption of CPOE by attending physicians varies widely. While placing a premium on the purchase of systems that meet the work‐flow needs of physicians and support the efficient entry of orders, hospital leaders can enhance physician adoption of this technology by communicating the role of CPOE in improving quality and safety.